Abstract

Decarbonisation requires an unprecedented deployment of variable renewable energy (VRE) in the form of solar photovoltaic and wind generation. As such, power systems worldwide are soon expected to experience, or indeed already experiencing high instantaneous penetrations of VRE. Given that these resources pose challenges but also present opportunities to the effective and efficient balancing of active power supply and demand in power systems, policy-makers worldwide are revisiting the design and configuration of power system operational practices such as grid codes, system operator processes and, in jurisdictions that have introduced wholesale competition, electricity markets.

In this thesis, I explore how policy-makers should design operational practices for balancing electricity markets given existing challenges and those posed by growing penetrations of VRE. Because the design process is complex, contested and contextual, I employ an empirical approach that identifies flexible and practical design solutions across operational timeframes (i.e. milliseconds to hours) by leveraging experience from the Australian National Electricity Market (NEM), the power system and electricity market that services the vast majority of the country’s electricity demand. The Australian NEM offers an excellent case study given that it regularly experiences high instantaneous penetrations of VRE, is increasingly integrating flexibility solutions in the form of demand-side and energy storage resources, and has transparent market arrangements that include a highly consultative market design process. Though I focus on the Australian NEM, I also provide policy-makers in other jurisdictions with valuable insights drawn from the Australian experience and a framework for approaching the design of operational balancing practices in electricity markets undergoing transition.

Frequency control services are a critical component to ensuring that imbalances are quickly addressed. In this thesis, I first explore what features are needed in centrally-coordinated arrangements for procuring frequency control services during energy transition. Through a comprehensive international review and an assessment of the Australian NEM’s frequency control arrangements, I offer four insights on designing frequency control arrangements as power system capabilities and needs change: 1) Understanding control action interactions, 2) Implementing efficient price formation and cost-allocation mechanisms, 3) Monitoring and assessing service provision to better align participant remuneration with service quality, and 4) Considering both regulatory and market mechanisms and their consequences and interactions.

Increased balancing flexibility will be required during energy transition. In wholesale electricity markets, market participation decisions determine the type and quantity of balancing flexibility available within scheduling timeframes. As such, market arrangements should be assessed to determine whether they are purpose-fit to deliver the balancing flexibility necessary to support high shares of VRE. As a contribution to this assessment, I investigate how balancing flexibility capabilities in scheduling timeframes are changing during energy transition. Using a practical method to quantify the time-varying spectrum of upwards and downwards flexibility capabilities of historical and projected resource mixes in two regions of the Australian NEM, I find that with high penetrations of VRE: 1) downwards flexibility margins can be exhausted around noon if wind and solar are unable or unwilling to provide it, 2) upwards flexibility becomes more scarce during morning and evening peak demand events and 3) a greater portion of upwards flexibility is provided by energy-limited resources. Based on these findings, I then recommend that policy-makers examine how existing operational practices can be augmented to elicit upwards flexibility provision, and that duration specifications and sustained footroom procurement be considered for reserve products.

Understanding the quantity of balancing flexibility available in a power system is inadequate if market participants are unable or unwilling to offer it into the wholesale spot market. As such, I explore whether market knowledge processes, which provide market participants with current and forecasted power system and market information, and participation rules in the Australian NEM are purpose-fit to enable the effective and efficient scheduling of energy storage resources – resources critical to balancing high renewables power systems through energy arbitrage. From an analysis of the centralised price forecasts generated by the NEM’s system and market operator, I find that errors in these forecasts are increasing in frequency and severity and, as such, arbitrage revenues can be reduced by 6 to ~50% should these forecasts be used to guide battery energy storage scheduling. I then investigate whether market participant (re)bidding is responsible for this phenomenon, and subsequently discuss changes to participant scheduling strategies and market design that could improve scheduling outcomes. I recommend that Australian policy-makers not only increase the frequency at which centralised knowledge processes are run, but also consider whether stricter market participation restrictions might incentivise participant bidding strategies that are less likely to induce sudden price forecast swings that can hamper effective scheduling.

The key contributions of the work contained within this thesis include the development, improvement and maintenance of open-source software that facilitates accessing and analysing data from the Australian NEM, detailed empirical analysis of frequency control, reserve and scheduling arrangements in the NEM and other restructured jurisdictions, and pragmatic market design and governance recommendations that could assist the NEM and other jurisdictions in maintaining a balance of power as they progressively retire fossil fuel generation whilst simultaneously deploying massive amounts of variable renewable energy.

Works arising from this thesis

Peer-reviewed journal articles

Prakash, A., Bruce, A. & MacGill, I. Insights on designing effective and efficient frequency control arrangements from the Australian National Electricity Market. Renewable and Sustainable Energy Reviews 161, 112303 (2022)

Prakash, A., Ashby, R., Bruce, A. & MacGill, I. Quantifying reserve capabilities for designing flexible electricity markets: An Australian case study with increasing penetrations of renewables. Energy Policy 177, 113551 (2023)

Prakash, A., Bruce, A. & MacGill, I. NEMSEER: A Python package for downloading and handling historical National Electricity Market forecast data produced by the Australian Energy Market Operator. Journal of Open Source Software 8, 5883 (2023)

Submitted manuscripts

Prakash, A., Bruce, A. & MacGill, I. The scheduling role of future pricing information in electricity markets with rising deployments of energy storage: an Australian National Electricity Market case study. Submitted to Energy Economics and under review

Regulatory submissions

Prakash, A., Keeratimahat, K., Bruce, A. & Macgill, I. Submission to the Semi Scheduled Generator Rule Change (s) Issues Paper. http://dx.doi.org/10.13140/RG.2.2.10054.50245 (2020)

MacGill, I., Prakash, A. & Bruce, A. Response to the Energy Security Board’s Post 2025 Market Design Consultation Paper. http://dx.doi.org/10.13140/RG.2.2.33241.75362 (2020)

Prakash, A., Macgill, I. & Bruce, A. Response to Frequency Control Rule Changes Directions Paper. http://dx.doi.org/10.13140/RG.2.2.11620.50560 (2021)

Prakash, A., Gorman, N., MacGill, I. & Bruce, A. Response to Reserve Services in the National Electricity Market Directions Paper. http://dx.doi.org/10.13140/RG.2.2.18331.39206 (2021)

Prakash, A., Ashby, R., Keeratimahat, K., Bruce, A. & MacGill, I. UNSW CEEM Response to Post 2025 Market Design Options Paper. http://dx.doi.org/10.13140/RG.2.2.27716.81284 (2021)

Prakash, A., Nicholls, A., Heim, D., Bruce, A. & Macgill, I. Response to Capacity Mechanism Project High-Level Design Paper. http://dx.doi.org/10.13140/RG.2.2.35007.59043 (2022)

Open source software development

Gorman, N., Prakash, A. NEMOSIS: NEM Open-source information service. A Python package for downloading historical data published by the Australian Energy Market Operator (AEMO). https://github.com/UNSW-CEEM/NEMOSIS (2023)

Prakash, A., Gorman, N., Katona, K., McConnell, D., Heim, D. NEMSEER: A package for downloading and handling forecasts for the National Electricity Market (NEM) from the Australian Energy Market Operator (AEMO). https://github.com/UNSW-CEEM/NEMSEER (2023)

Prakash, A. AEMO Monthly Data Archive Tool: A package and CLI utility to check for available data and download tables from AEMO’s Monthly Data Archive. https://github.com/prakaa/mms-monthly-cli (2023)

Acknowledgements

I wouldn’t describe myself as bold, but fortune has certainly favoured me.

I am lucky to have stumbled into doing a PhD with Anna and Iain as my supervisors. Thank you for taking me on, and for your wisdom, guidance and encouragement over the years. I will forever be grateful to you both for helping me embrace complexity and showing me the value of an interdisciplinary approach.

I am lucky to have met some amazing people within the Collaboration on Energy and Environmental Markets. Nick, thank you so much for our insightful chats, your incisive and constructive criticism of the work in this thesis, and of course, the whips and catches. I’ve learnt so much from your approach to software, problem-solving and life. To Sophie, thank you for our chats, the opportunity to work with you and for showing me the power of asking “why?”. To Simon, thanks for letting me always interrupt your days for our very enjoyable chats on campus. Rohan, thank you for letting me crash your honours project and giving me the opportunity to work with you. Declan, thanks for all our conversations about modelling, electricity markets and energy policy. Ash and Declan, thank you for signing up to write that submission with me. And to the broader group, thanks for your kindness and for sharing your wisdom and experience. I’ve enjoyed hearing about your work and sharing mine at all the discussion groups over the years.

I am lucky to have had some amazing friends who have helped me along the way whilst also tolerating my sub-par banter. Christian, thanks for being generous with your time in proof-reading and editing parts of this thesis. To Vicky, thanks for your love and support during the early years. To my various housemates – Nicola, Jeff, Hayden (!), Kate and Sammi in particular – thanks for being there through my highs and lows. Owen, thanks for showing me you can mix work with pleasure. To my trivia team and frisbee group, thanks for the very fun distractions. And to my many climbing partners, thank you for keeping me safe and sane. To everyone I couldn’t name here, I am lucky to be able to call you a friend.

I am lucky to have a mum and dad who have been very supportive and love me very much. Thank you for always being there for me and for reminding what is important in life.

I am lucky to have received feedback and support for the work in this thesis from many individuals and organisations. Thank you to Andrew Corrigan and Max Zekulich for sharing their data and analysis on frequency response and FCAS markets. To the team at WattClarity and to the Australian Energy Market Operator, the Australian Energy Market Commission and the Energy Security Board, thank you for the opportunity to present and get feedback on parts of this thesis. To the Australian Government Research Training Program and the UNSW Digital Grid Futures Institute, thank you for your financial support and to the UNSW Sydney Research Technology Services, thanks for giving me access to Katana. And to the various reviewers of the work in this thesis, thank you for your thoughtful comments.

Finally, I am lucky to have drawn the cards that I have in the lottery in life. I am immensely grateful to have been able to live, work and play on the awe-inspiring lands of the Dharug and Gundungurra people in a lucky country.

This thesis was written in Markdown and compiled with Pandoc using a template originally developed by Tom Pollard, with contributions from myself:

Tom Pollard et al. (2016). Template for writing a PhD thesis in Markdown. Zenodo. http://dx.doi.org/10.5281/zenodo.58490

1 List of Figures

2 List of Tables

Abbreviations and nomenclature

3 Introduction

3.1 Context and motivation

Mitigation of the wide-ranging, adverse impacts of climate change on ecological and human systems requires reducing and eventually halting global anthropogenic greenhouse gas emissions, of which largest source is the electricity sector (International Energy Agency, 2021; Pörtner et al., 2022). Decarbonisation demands an unprecedented deployment of renewable energy, storage and transmission to not only to replace emissions-intensive electricity generated from burning fossil fuels, but also to meet projected increases in the demand for electricity driven by economic development and electrification (International Renewable Energy Agency, 2020). Fortunately, investment in the energy transition is being bolstered by two reinforcing drivers: greater direct investment or support from policy-makers for low-carbon technologies and enablers, and declining technology and deployment costs – particularly for solar photovoltaic modules and lithium-ion batteries – thanks to experiential learning (Joskow, 2022; Roques, 2021; Way et al., 2022).

Consequently, power systems are currently experiencing or soon expected to experience high instantaneous penetrations of variable renewable energy (VRE), of which solar photovoltaic (PV) and wind generation are the most prevalent forms (Australian Energy Market Operator, 2019b; Matevosyan et al., 2021). The inherent variability, uncertainty and, in the case of inverter-based resources, asynchronicity of VRE can be problematic for balancing power systems (i.e. ensuring that active power supply and demand are more or less equal at each moment) that were originally designed to accommodate conventional (thermal and hydroelectric) generation (Ela et al., 2011; Kenyon et al., 2020). Though modelling has indicated that 100% renewable power systems can feasibly be balanced (Elliston et al., 2013; Hansen et al., 2019; Rey-Costa et al., 2023), these desktop studies assume that operational practices are appropriately configured to incentivise, control and/or coordinate power system resources such that they make sufficient balancing flexibility available to the system. This task – designing and configuring operational practices to ensure that high VRE power systems are successfully balanced – remains a significant and ongoing challenge that requires reassessing grid codes, system operator processes and electricity market design in jurisdictions that have introduced competition at the wholesale level (Papaefthymiou and Dragoon, 2016).

In this thesis, I explore how policy-makers should design operational practices for balancing electricity markets given existing challenges and those posed by growing penetrations of VRE. I do so within the context of the Australian National Electricity Market (NEM), the power system and electricity market that services ~80% of the country’s electricity consumption (Department of Climate Change, Energy, the Environment and Water, 2023).

3.2 Research gap

Whilst there is some degree of international consensus on high-level outcomes and the priority areas for designing operational balancing practices for electricity markets with growing penetrations of renewable energy, the multi-layered nature of the design problem, the presence of existing tensions and challenges, and a changing resource mix mean that the design process is complex and contested. Previous literature has emphasised the need for empirical research to identify feasible, flexible and contextually appropriate solutions, even if they may not be optimal. Through empirical studies of aspects of the Australian NEM, this thesis endeavours to address this knowledge gap in the Australian context. It also seeks to provide policy-makers in other jurisdictions with valuable insights drawn from the Australian experience and a framework for approaching the design of operational balancing practices in electricity markets undergoing transition.

3.3 Research framework

The research question of this thesis is outlined below:

Given existing design challenges and increasing penetrations of variable renewable energy resources as energy transition proceeds, how should we design operational practices for balancing electricity markets?

I focus on three aspects of this question that are encompassed by the following research objectives:

1. To determine what features are needed in centrally-coordinated arrangements for procuring frequency control services during energy transition.

2. To better understand how balancing flexibility capabilities in scheduling timeframes are changing during energy transition, and how these changes might impact the suitability and design of more decentralised operational balancing practices.

3. To explore how more decentralised operational balancing practices can be configured to maximise the deployability of balancing flexibility in scheduling timeframes.

I attempt to achieve these objectives through the following three research methods:

1. Review of academic and industry literature on the design of operational balancing practices from the Australian NEM and other jurisdictions.
2. Analysis of system and market data from the Australian NEM to provide empirical evidence for techno-economic modelling assumptions and to support the discussion of policy options.
3. Techno-economic (market) optimisation modelling. This includes the modelling of market regions of the NEM, and of market participation schedules for battery energy storage systems engaged in energy arbitrage.

The links between the research question, the research objectives and the research methods are shown in Figure 7 and are discussed in greater detail in Chapter 3.

3.4 Contributions

The most significant contributions of this thesis to addressing the research gap outlined in Section 1.2 include:

1. A review of the most prominent challenges to designing effective and efficient frequency control arrangements, and of the potential solutions to these challenges.
2. A comprehensive update to previous literature on frequency control in the NEM that includes recent experience with deteriorating frequency performance and the integration of VRE.
3. Arguments for delivering more robust frequency control arrangements that support investment in frequency control capabilities during energy transition.
4. A practical method to quantify the time-varying spectrum of system balancing flexibility capabilities for assessing, comparing and designing power system operational practices.
5. A case study that applies the method briefly described in Contribution 4 to test the sensitivity of balancing flexibility capabilities to conventional generation retirement, the rate of deployment of VRE and storage, resource mixes and operational constraints, and greater variability in operational demand.
6. A discussion of the role of balancing products in ensuring that sufficient balancing flexibility can be obtained from power system resources.
7. An analysis of historical centralised price forecast data from the NEM that highlights the increasing frequency and severity of errors in these forecasts.
8. Proposing a hypothesis that market participant (re)bidding may be partially responsible for the phenomena highlighted by Contribution 7.
9. Modelling the extent to which wholesale energy market arbitrage revenues might be reduced should the NEM’s centralised price forecasts be used to optimise battery energy storage scheduling.
10. A discussion of potential changes to market participant scheduling strategies and market design to improve energy storage resource scheduling outcomes.

Contributions 1–3 can be found in Chapter 4, which consists of a journal article published in Renewable and Sustainable Energy Reviews. Contributions 4–6 are included in a journal article published in Energy Policy that constitutes Chapter 5. Contributions 7–10 are reported in Chapter 6, which consists of a manuscript submitted to Energy Economics.

Additional contributions related to this thesis include:

• The development, improvement and maintenance of open-source software that facilitates accessing and analysing data from the Australian NEM (refer to the Works arising from this thesis and Appendix 12).
• The submission of several responses to Australian NEM rule change and market design consultations related to the topic of this thesis (refer to Works arising from this thesis).

3.5 Structure of the thesis

This thesis consists of 7 chapters and 4 appendices.

Chapter 2 provides relevant context on power systems and power system operation, and an overview of the literature on the desirable outcomes of and challenges associated with designing operational practices in electricity markets with growing penetrations of VRE.

Chapter 3 outlines the motivating research question, the research objectives and research methods of this thesis in detail.

Chapter 4 considers the question of how frequency control arrangements should be designed with growing penetrations of VRE. In this chapter, I first provide an overview of typical frequency control arrangements, with a focus on restructured electricity industries in North America and Europe, and the main challenges faced in their design. I then describe the NEM’s frequency control arrangements and the specific challenges posed by increasing penetrations of VRE. Based on an analysis of the performance of the NEM’s frequency control arrangements in responding to these challenges, I conclude this chapter by offering four key insights to policy-makers.

Chapter 5 focuses on understanding balancing flexibility capabilities available in scheduling timeframes as VRE and storage become a larger part of system resource mixes. In this chapter, I first provide an overview of how balancing flexibility is enabled and procured in the NEM before describing a methodology to quantify available reserves and footroom across deployment horizons for various resource types. I then quantify the available reserves and footroom in two regions of the NEM for existing resource mixes in 2020 and potential resources mixes in 2025, with two scenarios for the latter. From the findings of this case study, I explore the role of reserve products in securing balancing flexibility. Appendix 8 outlines the sources for key input data and assumptions, and provides further details regarding how these data were used in the analysis.

Chapter 6 explores how future pricing information and market participant operational strategies affect the deployability of balancing flexibility from energy storage resources. In this chapter, I first summarise market information, participation and clearing processes in the NEM in addition to providing context on grid-scale energy storage resource deployment, operation and market participation to date. Then, in a case study of the NEM, I examine errors in the NEM’s centralised price forecasts, propose a hypothesis to explain increasing divergence and the occurrence of price swings in these forecasts, and subsequently use these same forecasts to schedule a variety of battery energy storage systems for wholesale energy market arbitrage to assess the impact of imperfect foresight on arbitrage revenues. I conclude by discussing changes to market participant scheduling and market design that could maximise the balancing value of resources such as battery energy storage systems. Appendix 9 presents the mixed-integer linear program formulations used in the storage modelling in Chapter 6, and Appendix 10 describes the methodology used to model a storage scheduler discounting price forecasts (one of the formulations used in the storage modelling in Chapter 6 and described in Appendix 9). Appendix 11 describes the methodology used to adjust modelled raw reductions in arbitrage revenue potential to better reflect those that would be seen by an energy storage resource optimising its rebids in the NEM.

Chapter 7 concludes the thesis. In this chapter, I summarise the contributions of this thesis and highlight avenues for further work.

Appendix 12 is the Journal of Open Source Software article for NEMSEER, a Python package developed as a part of this thesis to download and handle historical NEM forecast data produced by the Australian Energy Market Operator.

4 Context and literature review

4.1 Introduction

In this chapter, I provide relevant context and a brief overview of the literature that tackles the challenge of designing operational practices in electricity markets with growing penetrations of variable renewable energy (VRE).

As I summarise in Section 2.6, this review highlights a role for empirical work that identifies “second-best” and flexible design solutions given the specific context of each power system and jurisdiction. This approach to designing operational balancing practices constitutes a knowledge gap that I aim to address in the Australian context within this thesis (see Section 3, the research framework for this thesis).

The rest of the chapter is structured as follows. Firstly, I provide a brief description of power systems in Section 2.2. Then, in Section 2.3, I present a high-level overview of power system phenomena and practices in operational timeframes, highlight the importance of active power balancing and discuss power system operational paradigms with a focus on restructured electricity markets. Following this, in Section 2.4, I describe existing and emerging balancing practices in operational timeframes and, in Section 2.5, I propose good design outcomes and discuss the challenges involved in operational practice design. Finally, in Section 2.6, I summarise the motivation for the work contained within this thesis.

In this chapter, I use bold text to introduce definitions for a (new) term or concept.

4.2 Power systems

Given the welfare and economic benefits associated with electricity access, many countries in the 20^th century constructed bulk power systems to leverage investment and operational economies of scale. These systems sought to efficiently deliver active power (the component of apparent power that does work at a load) to numerous electricity end-users (in the aggregate, system demand or load) from electricity suppliers (generators) across vast distances. A typical power system configuration is presented in Figure 1. Generators supply the system with alternating current (AC) power either through a direct electromagnetic connection or, if they are inverter-based resources (IBRs)¹, through a power electronic inverter interface that converts the direct current (DC) power produced by the generator to AC power. AC power is then transmitted over long distances through a high voltage transmission system. As transmission lines approach load centres, voltages are stepped down to make power delivery to the majority of end-users connected to the lower voltage distribution system safer (Masters, 2004).

4.2.1 Synchronous and control areas

A network area that is operated at a (constant) nominal AC frequency is known as a synchronous area. During normal operation, AC frequency should be close to the system’s nominal value (see Section 2.3.2.1 as to why) and more or less uniform across the synchronous area. A control area, on the other hand, is a network area which a particular system operator (SO) is responsible for operating. In this thesis, I use the term jurisdiction interchangeably with control area, with a preference for the former when referring to a control area with a wholesale electricity market overlay.

Whether the term “power system” refers to a synchronous area or a control area is dependent on context and the relationship between the two in the jurisdiction in question. In eastern and southern Australia, the National Electricity Market’s (NEM) single control area consists of two synchronous areas (see Section 4.5 for further detail). In contrast, other jurisdictions have a single synchronous area composed of several electrically-connected control areas demarcated by political, rather than physical boundaries. For example, continental Europe is a single synchronous area consisting of many national or trans-national control areas, and the continental United States has three synchronous areas (two of which extend into Canada) with over 60 control areas (North American Electric Reliability Corporation, 2023; Schittekatte and Pototschnig, 2022).

4.3 Power system operations

Operating a power system involves the direction or control of power system resources – generators, loads, network elements and energy storage resources. In most cases, power system operation is an economic optimisation problem that aims to minimise system costs (or under some market paradigms, maximise the value of trade) whilst 1) continuously maintaining a balance between active power supply and demand and 2) ensuring that system resources and the system itself are operated within their respective technical envelopes (Wood et al., 2014). The former constraint more or less corresponds to reliable operation², and the latter implies secure (or stable) operation and is a prerequisite for reliable operation (Andersson, 2021). Maintaining a secure and reliable power system is vital; restarting the system after failure (system restoration) is a long and complex procedure, and power outages (blackouts), whether they be localised or across a wider area, can have devastating social and economic consequences (Kirschen and Strbac, 2004).

Figure 2 presents a high-level overview of power system phenomena in operational timeframes and common operational practices (i.e. processes, services and markets in operational timeframes). The processes, services and markets discussed in detail within this thesis are highlighted in bold red text. Though processes, services and markets in investment and planning timeframes are not within the scope of this thesis, it is important to acknowledge that they can influence the manner in which a power system is operated.

4.3.1 Phenomena in operational timeframes

A knowledge of physical power system phenomena and the timescales in which they occur is an important prerequisite to achieving good outcomes when designing operational practices for power systems. As shown in Figure 2, power system operations is concerned with three phenomena that dominate on timescales ranging from a few milliseconds to several months (Hatziargyriou et al., 2021; Machowski et al., 2020):

1. Electromagnetic phenomena arise from the coupling of electrical and magnetic fields within synchronous machines (generators and motors that rotate at a speed proportional to AC frequency) and between power system resources. They occur on the timescale of milliseconds to seconds. IBR controls also operate in this timeframe.

2. Electromechanical phenomena are slower (seconds to minutes) and arise as a result of electromagnetic fields interacting with rotating masses and mechanical forces. These typically occur in generators and motors.

3. Thermodynamic phenomena are slower still. They encompass chemical fuel conversion and heat transfer processes in boilers. These phenomena occur over seconds to minutes to hours. The dynamics of the primary energy sources for hydroelectricity and VRE are also studied across these timescales (Keeratimahat et al., 2021).

4.3.2 Active power balancing

Active power balancing can be described in simple terms using the law of conservation of energy: the energy supplied into a network node through primary energy conversion, energy storage or transmission is equal to the sum of the energy dissipated, stored, consumed and transmitted away from the same network node at each and every moment. A more practical definition that encompasses the key concerns of power system engineers is that active power balancing requires real-time control of generation and loads to balance active power supply and demand across the power system:

• Real-time (i.e. moment-to-moment) control of supply and demand is required because it is still uneconomical in many jurisdictions to store electricity at scale despite grid-scale storage cost reductions (International Energy Agency, 2022). In other words, many power systems have small balancing buffers and thus require real-time balancing.
• Whilst electricity can be transported close to the speed to light across a synchronous area, some degree of balancing coordination across the power system is required because of transmission losses and constraints on network flows imposed by line thermal limits, stability requirements and Kirchoff’s circuit laws (Hirth et al., 2016; Kirschen and Strbac, 2004). A whole-of-system approach is also necessary because active power imbalances in network sub-regions can have system-wide consequences (see Section 2.3.2.1).

4.3.2.1 Why is balancing required?

Unlike many other commodity transportation networks, an active power supply-demand imbalance can lead to deviations in technical parameters – voltage and AC frequency – that not only have the potential to damage equipment connected to the power system, but also to trigger a system collapse (see Section 2.3.2.1.2) (Borenstein et al., 2023).

4.3.2.1.1 The relationship between active power balance and AC frequency

The presence of synchronous machines in most power systems means that system active power balance is closely tied to the system’s AC frequency. During stable operation, synchronous machines rotate at a synchronous speed (N_s) that is proportional to the power system frequency (f) (Equation 1) (Chapman, 2011):

N_s = \frac{120f}{P}\qquad{(1)} where N_s is the synchronous speed in revolutions per minute, P is the number of (rotor) magnetic poles and f is the electrical frequency in Hertz (Hz).

Equation 1 describes an important characteristic of synchronous machines during stable operation, but the link between active power balance and power system frequency requires examining synchronous machine dynamics. As shown in Figure 3, the interaction between the magnetic fields of the rotor and stator of a synchronous generator (e.g. coal-fired, gas-fired and hydro generation) produces an electromagnetic torque (T_e) on the rotor that opposes the mechanical torque (T_m) supplied by a prime mover (e.g. steam turbine). Equation 2, which is a variation of what is known as the swing equation, is an energy balance across the synchronous generator. It shows that if there is a transient increase in the electrical load of the power system (equivalent to an increase in P_e and thus T_e), the rotor of a synchronous generator will begin to decelerate as its stored kinetic energy is converted to electrical energy (Elgerd, 1971; Grainger, 1994).

J\omega_{sm}\frac{d\omega_{sm}}{dt} = P_m - P_e\qquad{(2)} where \omega_{sm} is the synchronous machine rotor shaft velocity, J is moment of inertia of the rotor, P_m is mechanical power, T_m is mechanical torque, P_e is electrical power and T_e is electromagnetic torque.

The relationship between the active power imbalance in a power system (P_{gen}-P_{load}) and AC frequency is obtained by extending the swing equation from a single synchronous generator to all synchronous generators in a synchronous area (Equation 3). Equation 3 shows that the rate of change of frequency (RoCoF) is proportional to the active power imbalance and inversely proportional to the system’s inertia constant, H. This form of the swing equation only models the inertial response of synchronous generators; that is, it does not include the load damping response offered by (frequency-dependent) induction motor loads. The generation inertia constant is often used as a proxy for the system inertia constant since the high speed and mass of generator rotors mean that they store significant quantities of kinetic energy (Denholm et al., 2020; Ulbig et al., 2014).

\frac{2H}{f}\frac{df}{dt} = \frac{P_{gen}-P_{load}}{S_{g, total}}\qquad{(3)} where H is the inertia constant of the synchronous area (H=\sum_{g} H_g, where H_g = \frac{J_g(2\pi f)^2}{2S_g}), f is the AC frequency, \frac{df}{dt} is the rate of change of frequency or RoCoF, S_{g,total} is the total apparent power of synchronous generators, and P_{gen} and P_{load} are the power system’s total active power supply and active power demand (including losses), respectively.

Equation 3 shows that a power system’s AC frequency is an indicator of active power balance (Baggini, 2008). Insufficient generation will lead to a decrease in system frequency (i.e. negative RoCoF) and oversupply will lead to an increase in system frequency (i.e. positive RoCoF).

4.3.2.1.2 The consequences of frequency deviations

Serious power system frequency deviations away from the nominal value can have harmful effects. Synchronous machines may experience equipment-damaging vibrations (Ulbig et al., 2014), and both synchronous machines and transformers can overheat and fail if they operate outside their rated voltage-frequency limits (Kirby et al., 2002). Synchronous machines are also vulnerable to damage from high RoCoFs due to pole slipping (DGA Consulting, 2016). For these reasons, frequency-sensitive relays are often used to protect power system resources from frequency excursions.

However, these same equipment protection measures can also trigger the complete collapse of the power system. Should the disconnection of a resource following a relay trip exacerbate an existing active power imbalance, the system frequency may deviate further and result in further disconnections. Situations such as these are known as cascading failures and can lead to the collapse of the entire power system. As such, SOs often employ emergency frequency control schemes to arrest imbalances by tripping loads in the event of under-frequency (under-frequency load shedding or UFLS) or generation in the event of over-frequency (over-frequency generation shedding or OFGS) (Australian Energy Market Operator, 2021a; Hartmann et al., 2019). The activation of these schemes is undesirable, particularly since UFLS can lead to power system reliability standards being breached.

4.3.2.2 Threats to active power balance

Threats to active power balance can be broadly categorised as either power system variability or power system uncertainty.

4.3.2.2.1 Power system variability

Power system variability refers to expected or forecasted changes to active power supply and/or demand. Sources of variability include fluctuations in load, oscillatory active power output from synchronous generators and VRE generation ramping (i.e. a sustained increase or decrease in active power output). VRE ramping includes changes in solar PV generation during sunrise or sunset and in wind generation with wind speed variations (Australian Energy Market Operator, 2020d; Bloom et al., 2017; Ela et al., 2011).

4.3.2.2.2 Power system uncertainty

Power system uncertainty refers to unexpected changes to active power supply and/or demand. Sources of uncertainty include demand and VRE generation forecast errors, and singular or widespread outage events triggered by the weather or unexpected system responses and interactions (Australian Energy Market Operator, 2020d; Eggleston et al., 2021; Ela et al., 2011).

4.3.3 Operational paradigms

Though governance and operational arrangements vary from jurisdiction to jurisdiction, the powers, responsibilities and degree of ring-fencing imposed upon the SO are largely dictated by one of two possible operational paradigms for a control area (Chawla and Pollitt, 2013): where the SO is a vertically-integrated utility, and where the SO is, at the very least, responsible for operating a transmission system that forms the physical basis of a wholesale electricity market. In both cases, it is the SO that is ultimately responsible for ensuring that the transmission network in their control area is operated in a secure and reliable manner (Roques, 2008).

4.3.3.1 Vertically-integrated utility

Under this paradigm, a single company (either state-owned or privately-owned but regulated) owns, operates and invests in generation, transmission and distribution infrastructure, as well as being responsible for the retail of electricity to the end-user. This was the sole operational paradigm for much of the 20^th century. Having a single owner and operator of power system resources reduces complexity and transaction costs, and enables economies of scale in both asset investment (particularly generation infrastructure) and operation (Sioshansi, 2006). The benefits from economies of scale are material where industrialisation and/or electrification are driving sustained load growth. This was the case in advanced economies in the 20^th century and is still the case in many emerging economies (Hogan, 2008; Roques and Finon, 2017).

4.3.3.2 Wholesale electricity markets

Beginning in the late 1980s, some jurisdictions opted to restructure their electricity sector. To varying degrees across different jurisdictions, the impetuses for restructuring included advancements in small low-upfront cost gas turbine technologies, the promise of consumer choice, perceptions that vertically-integrated utilities were inefficient and politicised, and a political zeitgeist prevalent at the time that pursued economic efficiency through privatisation and competition (Chester and Elliot, 2019; MacGill and Healy, 2013; Simshauser, 2019; Sioshansi, 2006). Two features common to electricity industry restructuring processes were the unbundling of vertically-integrated utilities and the introduction of competition for wholesale supply (and in some cases, demand) via an electricity market – an auction-based mechanism for the sale and/or purchase of electrical energy.

4.3.3.2.1 Unbundling

In most cases, the unbundling of a vertically-integrated utility divided generation ownership and barred the SO from owning generation assets. In some jurisdictions, SOs retained ownership of the transmission network (e.g. Transmission System Operators, or TSOs, in many European control areas) whereas others made their SOs “independent” by relieving them of any asset ownership (e.g. Independent System Operators in North American control areas). Some SOs, such as those in North America and the Australian NEM, were also given market operation responsibilities (Chawla and Pollitt, 2013).

4.3.3.2.2 Market models

The design and implementation of wholesale electricity markets differs across jurisdictions that undertook electricity industry restructuring. Despite these differences, electricity markets worldwide can broadly be categorised into two markets models³ that are distinguished by the degree of centralisation in system and market operations (Ahlqvist et al., 2022; Barroso et al., 2005; Cramton, 2017):

1. Central dispatch markets, where decisions regarding dispatch and, in some cases, unit commitment (see Section 2.4.2) are made by the SO. System and market operations are often integrated (i.e. the SO is also the market operator) through a mandatory power pool in which supply offers are aggregated and cleared against a demand forecast (one-sided pool), or against an aggregated demand curve constructed from potential buyers (two-sided pool) (Barroso et al., 2005). In these markets, locational prices for energy and ancillary services (services procured to maintain security and reliability) are produced by SO-run centralised optimisation processes that consider the physical constraints of the transmission system. This market model has been adopted in Independent System Operator/Regional Transmission Operator (ISO/RTO) markets in North America (refer to Section 4.4.2.1 for more detail). As I discuss further in Section 6.4.1, the Australian NEM’s design is predominantly based on this model though it does incorporate some features of more decentralised markets.

2. Decentralised or self-dispatch markets, where decisions regarding dispatch and unit commitment are made by market participants, and in which system and market operations are more decoupled. These types of markets facilitate trade through bilateral contracts between suppliers and buyers. Whilst scheduling and dispatch is managed by market participants, they are required to submit intended schedules to the SO ahead of delivery (often during the day before delivery). The SO is responsible for taking redispatch actions to ensure that transmission constraints are not violated, and for determining the requirement for and procuring balancing services (another name for frequency control services, which I discuss in greater detail in Section 2.4.1) that maintain system balance following market gate closure. As outlined in Section 4.4.2.2, this is the dominant market model in Europe.

Figure 4 shows the primary and secondary commercial arrangements in each of these market models. A mandatory power pool is the primary exchange mechanism in central dispatch markets, whereas self-dispatch markets are designed to facilitate exchange through bilateral contracts. However, both exchange mechanisms are present in each market model. Bilateral contracts (in the form of derivatives) are often used as hedging instruments in central dispatch markets, and several self-dispatch markets, such as those in Europe, have associated voluntary power exchanges that are essentially power pools (Barroso et al., 2005).

4.3.3.2.3 Market platforms

Power system resource inflexibilities and the desire for physical and financial risk management mechanisms in operational timeframes have driven policy-makers in many jurisdictions to design and implement electricity markets with multiple market platforms (Energy Security Board, 2020a; Isemonger, 2006). Platforms are formal sub-markets for energy (and sometimes ancillary services) that are cleared ahead of the delivery of electricity and/or ancillary services. The number of platforms implemented in a particular market is often related to its market model. Self-dispatch markets can maximise trade and better facilitate market participants balancing their positions by implementing multiple market platforms (typically day-ahead and several intra-day, see Section 4.4.2.2), whereas the number of platforms in central dispatch markets (typically real-time and in most cases, day-ahead) is limited by the computational complexity of the optimisation algorithm(s) used by the SO to clear each market platform (see Section 4.4.2.1 and Section 4.5) (Ahlqvist et al., 2018).

4.4 Balancing practices in operational timeframes

SOs employ balancing practices in operational timeframes (including the non-faded processes, services and markets shown in Figure 2) to obtain balancing flexibility. Balancing flexibility is procured either to address variability directly, or as optionality to manage uncertainty (Heggarty et al., 2020, 2020; Papaefthymiou and Dragoon, 2016). Though the particularities of these practices vary between jurisdictions, they are almost always organised in a hierarchical and sequential fashion to ensure that active power supply and demand are continuously balanced across different timeframes. Furthermore, in jurisdictions that have restructured their electricity industries, balancing practices that were previously administered by a vertically-integrated utility have been adapted into or integrated with market-based mechanisms.

In the subsections that follow, I describe balancing practices in the order of the timescales in which they are relevant. It is important to stress that a particular jurisdiction may formally incorporate only a subset of the practices described below in their operational procedures and market design.

4.4.1 Frequency control services

Frequency control services (leftmost section of the processes, services and markets shown in Figure 2) are ancillary services used by the SO to contain AC frequency within as narrow a band as possible during normal operation and following contingency events (sudden disturbances) (Eto et al., 2018). With the exception of inertial response from synchronous machines (Section 2.4.1.1), these services are provided by power system resources with 1) the appropriate control system configurations and 2) capacity flexibility (or reserves) in the form of headroom (the ability to increase active power output) for responding to an under-frequency event and/or footroom (the ability to decrease active power output) for responding to an over-frequency event (Eto et al., 2010). Vertically-integrated utilities schedule their own resources to provide frequency control services, whereas SOs in restructured electricity industries typically procure frequency control services through regulatory and market-based mechanisms (see Section 4.4.2).

As shown in Figure 5 and discussed further in Chapter 4, the conventional frequency control services described below differ based on their purpose, response time and activation and control methods.

4.4.1.1 Inertial response

As discussed in Section 2.3.2.1.1, synchronous machines have an inherent inertial response to AC frequency deviations that must be considered in the frequency control strategy of a power system. For a given active power imbalance, the inertia constant of the synchronous area (H in Equation 3) determines the magnitude of the initial RoCoF following an imbalance event and the speed at which the power system can be returned to its nominal frequency (Tielens and Van Hertem, 2016; Ulbig et al., 2014).

4.4.1.2 Fast frequency response

IBRs and loads on frequency-responsive relays can provide what is typically known as fast frequency response (FFR). The most well-discussed use-case for FFR is mitigating high RoCoFs through a response delivered within a matter of milliseconds to a few seconds following a contingency event (Australian Energy Market Operator, 2017a; Miller et al., 2017b). As I also touch upon in Section 4.4.3.1, the term FFR has been used rather loosely to date to refer to three distinct control configurations:

1. An inherent response delivered by IBRs that, though lacking a spinning mass, resembles the inertial response of synchronous machines (sometimes referred to as virtual inertia) (Lin et al., 2020);
2. A controlled response delivered by wind generation in which kinetic energy is extracted from a wind turbine rotor to rapidly inject active power into the system (sometimes referred to as synthetic inertia or inertia-based FFR) (Eriksson et al., 2018; NERC Inverter-Based Resource Performance Task Force, 2020);
3. A controlled and sustained response delivered by IBRs and frequency-responsive loads that is more or less a faster version of primary frequency response (Section 2.4.1.3) (Dreidy et al., 2017; Fernández-Guillamón et al., 2019; NERC Inverter-Based Resource Performance Task Force, 2020).

4.4.1.3 Primary frequency response

The aim of primary frequency response (PFR) is to arrest a frequency deviation. PFR is implemented in resource-level control systems such that each enabled resource provides a response to locally-measured frequency deviations that exceed a certain control dead-band (Ela et al., 2012b; Wang and Hiskens, 2003). For generators, this is achieved through droop control, in which a synchronous speed deviation produces a change in the active power output of a generator according to its droop characteristic (Figure 6, e.g. from A to B along L_0). Droop control is implemented in the turbine governors of synchronous generators and the inverter control systems of IBRs (Fernández-Guillamón et al., 2019; Lin et al., 2020). Provided there is a sufficient amount of PFR reserve to arrest the system frequency, the frequency zenith/nadir (maximum/minimum system frequency following an active power imbalance event) is determined by the size of the initial imbalance, the inertia constant of the synchronous area, the droop characteristics of power system resources and the speed of PFR (NERC Inverter-Based Resource Performance Task Force, 2020). PFR should ideally be sustained until secondary frequency control can take over (i.e. several to tens of seconds) (Eto et al., 2010; Eto et al., 2018; Undrill, 2019, 2018).

4.4.1.4 Secondary frequency control

Secondary frequency response (SFR) is designed to relieve fast-acting PFR. SFR is implemented in resource-level load controllers, which can (Eto et al., 2018; Undrill, 2019).:

1. Be pre-configured to respond following a frequency deviation through a frequency bias setting. This could include sustaining already-delivered PFR (as shown in Figure 6); or
2. Receive control signals from Automatic Generation Control (AGC), a control system used by the SO to coordinate SFR across the control area. The AGC’s control objective is to minimise Area Control Error subject to a tie-line bias and thus return power system frequency to its nominal value. Following the calculation of a required response that occurs in each cycle (these are typically several seconds apart), the AGC then communicates with each enabled resources to provide them with active power adjustment targets (Machowski et al., 2020). The service provided by these enabled resources is referred to as regulation in many jurisdictions (Ela et al., 2011; Hewicker et al., 2020).

4.4.1.5 Tertiary frequency control

In power systems where scheduling processes are infrequently run (e.g. vertically-integrated utilities that historically produced hourly schedules) or in which a “safety margin” is desired to address active power imbalances that endure over multiple scheduling intervals, tertiary frequency response (TFR) is deployed to relieve PFR and SFR (Hewicker et al., 2020). Some jurisdictions, such as those operated by the California and Midcontinent ISOs, have introduced ramping reserves – a form of TFR intended to address increased variability and uncertainty across dispatch intervals (i.e. several minutes to an hour) due to growing penetrations of VRE (Ela et al., 2017; Ela et al., 2016). Other jurisdictions, such as the Australian NEM, rely on balancing flexibility obtained through frequent scheduling processes (though the introduction of a TFR service had recently been proposed; see Section 5.4.3.1) (Australian Energy Market Operator, 2020a; Riesz et al., 2015).

4.4.2 Scheduling

The purpose of scheduling is to produce efficient (or economic) generation and consumption schedules for the minutes to days ahead based on expected power system conditions. In a similar manner to Chow et al. (2005), I divide the scheduling problem into three phases: dispatch, unit commitment and longer-term scheduling.

4.4.2.1 Dispatch

Dispatch involves assigning generation or consumption targets to already-committed power system resources in real-time (i.e. several minutes ahead of delivery). Dispatch is carried out by the monopoly utility in vertically-integrated electricity industries, the SO in central dispatch markets and is self-managed by market participants in self-dispatch markets. In the first two cases, the SO dispatches power system resources by running a process known as security-constrained economic dispatch. Security-constrained economic dispatch seeks to find a minimum cost operating configuration for committed generation and loads such that a short-term forecast of non-scheduled demand can be met subject to network constraints and stability and reliability requirements⁴ (Grainger, 1994). Some SOs solve this problem for a single interval (e.g. in the Australian NEM), whereas others, including the California and Midcontinent ISOs, solve a multi-period dispatch to procure and, to some extent, price capabilities to address expected non-scheduled demand ramps (Ela and O’Malley, 2016; Schiro, 2017). The dispatch solution for each dispatch interval (typically 5–15 minutes long) consists of generation and consumption setpoints, enablement quantities for resources providing frequency control services and, in central dispatch markets that integrate power system and market operation, real-time market locational marginal prices for energy and ancillary services (Cramton, 2017). If piecewise linear functions are used by vertically-integrated utilities to model resource cost curves, or are required by the real-time market bid format for a market participant’s energy offer curve⁵, the security-constrained economic dispatch problem can be efficiently solved using linear programming techniques (Wood et al., 2014).

4.4.2.2 Unit commitment

Thermal and hydroelectric generation, which historically dominated supply in many power systems, have inflexibility constraints (minimum load, start-up time, ramping limits and minimum up and down times) and costs (those attached to resource start-up, shut-down and operation at minimum load) that require SOs and market participants to make non-trivial unit commitment decisions (i.e. whether a resource should be online or offline). Depending on the resource, these decisions are made several minutes to hours ahead of power delivery (Agora Energiewende, 2017; Denholm et al., 2018). Unit commitment is:

• Managed by vertically-integrated utilities for all resources in jurisdictions that have not undergone restructuring;
• Built into the day-ahead market and intra-day reliability processes in central dispatch markets; and
• Self-managed by market participants in self-dispatch markets and in single-platform semi-centralised markets such as the Australian NEM.

In the first two cases, the SO runs a process known as security-constrained unit commitment, which seeks to determine the minimum cost subset of power system resources that should be committed to meet a non-scheduled demand forecast for a future horizon (usually 36–48 hours ahead) subject to network constraints and stability and reliability requirements. Security-constrained unit commitment is usually formulated as a mixed-integer linear program. Solving mixed-integer programs is computationally complex due to the non-convexity of the integral solution space (Knueven et al., 2020; Wood et al., 2014).

In the day-ahead platforms of central dispatch markets, market participants submit start-up and minimum load costs in addition to a piecewise linear offer for energy (Herrero et al., 2020). The SO then solves a security-constrained unit commitment problem to clear the day-ahead market. This produces locational prices for energy and ancillary services for each market interval in the day-ahead horizon (usually each hour) and an ahead schedule (Cramton, 2017; Isemonger, 2009). The ahead schedule is typically only financially binding; that is, any real-time deviations from this schedule are settled using real-time market prices. As discussed in Section 2.3.3.2.3, the day-ahead market platform provides market participants with an opportunity to hedge their real-time market position, and gives both market participants and the SO greater certainty of resource schedules before power delivery.

4.4.2.3 Longer-term scheduling

Operational planning actions taken in longer-term scheduling timeframes (i.e. a day to years ahead) include resource maintenance scheduling, the management of energy/fuel reserves and ensuring that any social and environmental obligations placed on resources are met (e.g. regulated discharges from hydroelectric scheme dams). Many of these activities are conducted on the basis of information supplied by longer-term weather/climate, power system and market forecasts (Denholm et al., 2018; Helistö et al., 2019; Suckling, 2018). Energy reserve management is a particularly important aspect of longer-term scheduling for power system resources that face material opportunity-costs due to limited energy/fuel storage capacity, seasonally-variable primary energy source availability and/or degradation from operation (McPherson et al., 2020; Xu, 2022). In restructured electricity industries, longer-term scheduling also requires market participants to consider and potentially modify their position in forward markets in which electricity derivatives and contracts are traded (MacGill and Esplin, 2020).

4.5 Designing balancing practices in operational timeframes

Energy transition has prompted policy-makers worldwide to revisit and redesign existing balancing practices in their jurisdictions. There is some degree of international consensus surrounding desirable high-level design outcomes (Section 2.5.1) and a suite of changes that could assist with accommodating greater shares of VRE. I describe these changes in Section 4.4.3.1 and Section 5.3, but they most notably include enabling greater demand-side participation, making electricity markets faster and more flexible, and acquiring more balancing flexibility by modifying existing reserve products and introducing new ones (Holttinen et al., 2021; Papaefthymiou and Dragoon, 2016; Riesz and Milligan, 2015).

However, various barriers and complexities pose challenges to the design and implementation of such mechanisms and contribute to the contested nature of the design process (MacGill and Esplin, 2020; Papaefthymiou et al., 2018; Schittekatte and Meeus, 2020; Silva-Rodriguez et al., 2022). I discuss the most pertinent of these in Section 2.5.2.

4.5.1 Outcomes of good design

Below, I present three desirable outcomes of the design process that I use to assess changes to balancing practices throughout this thesis. All three have been previously discussed in the literature and in a co-authored submission to an Australian NEM reform process (MacGill et al., 2020b). As I discuss further in Section 4.4.3 and Section 5.3, in practice there are trade-offs that mean that an improvement in one outcome may come at the expense of another.

1. Effectiveness. Effective balancing practices procure balancing flexibility that is sufficient, in terms of both quantity and performance, to ensure that power system balancing requirements are met. I extend effectiveness to also include the robustness of balancing practices to the wide range of future operating conditions and system configurations that may arise as energy transition proceeds (Australian Energy Market Operator, 2021a; Prakash et al., 2021).

2. Efficiency. Efficient balancing practices procure balancing flexibility at the lowest cost to the system, both now (productive efficiency) and into the future (dynamic efficiency). Furthermore, efficient arrangements should also procure the right mix of balancing flexibility according to balancing requirements determined by user and/or system needs (allocative efficiency).

3. Minimising administrative costs and complexity. Administrative costs, such as those associated with operating a market for procuring balancing flexibility or with verifying its delivery, can be significant. Administrative costs include expenses related to metering equipment, IT systems and additional staff. Complex administrative arrangements can be problematic as they may hamper effective balancing flexibility delivery and/or interact with other power system processes in ways that are unforeseen and unintended.

4.5.2 Existing and emerging challenges in the design process

4.5.2.1 Variable renewable energy and inverter-based resources

Many jurisdictions are presently experiencing or are soon expected to experience high instantaneous penetrations of VRE resources and IBRs (Australian Energy Market Operator, 2019b; Ela et al., 2021; Matevosyan et al., 2021). VRE resources pose challenges to power system balancing as they introduce additional variability and uncertainty, and because IBRs do not provide an inherent or controlled response to frequency deviations unless they are explicitly configured to do so. I elaborate on these challenges in Section 4.3, Section 4.4.3.1 and Section 5.3. These challenges are of greater concern to islanded power systems and weakly-interconnected control areas that have limited to no access to a wider synchronous area for balancing assistance (Hodge et al., 2020).

4.5.2.2 The tension between effectiveness and efficiency

The tension between effectiveness, the primary objective of engineering standards and practices, and efficiency, an outcome championed by economists and often pursued through markets, significantly contributes to the complexity and contested nature of the design process in restructured electricity industries. One perspective of this tension which I present in Section 4.3, Section 5.3 and is extensively discussed by Chao et al. (2005)) is that the efficiency gains obtained through market-based mechanisms for procuring balancing flexibility may come at the expense of the redundancy, certainty and control that a SO might require to guarantee effective balancing. An alternative perspective of this tension is that restructuring enables market participants to scrutinise and lobby for changes to operational practices that adhere to engineering best-practice regardless of the costs of doing so. In other words, market participants can help achieve efficient operation through advocacy “because choices of operating standards can severely impact their profits”, especially in the case of practices associated with “costs that the [SO] passes to participants via grid management charges” (Chao et al., 2005, p. 1984).

The tension between effectiveness and efficiency can be reframed as the problem of specifying the degree to which balancing flexibility procurement arrangements are centralised (i.e. managed by the SO). The SO is best placed to manage the procurement of specialised and system-critical frequency control services, whereas the provision of some forms of balancing flexibility in scheduling timeframes can largely be left to market participants to self-manage. I use this problem framing when outlining the research objectives of this thesis in Chapter 3.

4.5.2.3 The requirement for “second-best” design

Perspectives from both engineering and economics have been invoked in the electricity market design literature when referring to the “second-best” design challenge. MacGill and Esplin (2020) approach the market design problem with a systems engineering perspective guided by the principle of sub-optimisation, which holds that:

Optimizing each subsystem independently will not in general lead to a system optimum, or more strongly, improvement of a particular subsystem may actually worsen the overall system (Machol, 1965)

Pollitt and Anaya (2019) and Mays (2021) discuss a similar challenge, but instead invoke Lipsey’s “General Theory of Second Best”, which states that (Lipsey and Lancaster, 1956, p. 11):

If there is introduced into a general equilibrium system a constraint which prevents the attainment of one of the Paretian conditions, the other Paretian conditions, although still attainable, are, in general, no longer desirable

In other words, if a constraint imposed by technical, social or political factors (some of these are referred to as “market distortions” in the literature) or by the construct of the market itself (discussed later in this subsection) prevents some conditions and/or features that are necessary for an efficient outcome, then 1) pursuing the other conditions and/or features required by that design does not guarantee an improvement in system welfare (it may, in some cases, even worsen it) and 2) imposing other “market distortions” (often regulatory mechanisms) on the system may actually improve welfare outcomes. For example, “the adoption of a free trade policy by one country, in a multi-country tariff ridden world, may actually lower the real income of that country and of the world” (Lipsey and Lancaster, 1956, p. 14).

Several facets of the design problem warrant a “second-best” approach to design solutions. Firstly, as raised in Section 4.3 and modelled in Nahmmacher et al. (2016), there is often an asymmetry between the high costs⁶ of power system collapse due to inadequate balancing flexibility, and the relatively lower costs of measures that the SO can take to mitigate system failure (Lal et al., 2021). This cost asymmetry suggests that there are externalities tied to security and reliability (security-of-supply) that are not reflected in the price of balancing processes, services and products (Keppler et al., 2022; MacGill and Esplin, 2020). As suggested by R. H. Coase’s analysis of the problem of “social cost”, there exists a bargaining solution for addressing security-of-supply externalities in the form of differentiated reliability (i.e. consumers of electricity pay for different levels of service) (Billimoria et al., 2020; Mays et al., 2022); however, the issue of equitable access to an essential service and the significant transaction costs⁷ associated with this solution have led to the characterisation of security-of-supply as a public good and the allocation of responsibility for reliability to the SO (Coase, 1960). The ultimate outcome here is that the effective provision of the public good (i.e. meeting a system-wide requirement for reliability) requires “distortions” in the markets for the products and services that are required to provide it (Mays, 2021). For example, many markets for frequency control services are monopsonistic; the SO is the sole buyer and controls market demand, albeit on behalf of the system to ensure that sufficient capabilities are procured for secure and reliable operation (Y. Rebours et al., 2007). I discuss the role of other “distortions” and market incompleteness in hindering efficient price formation and cost allocation in Australian frequency control markets in Chapter 4.

Secondly, because unbundled SOs have neither direct control nor ownership of generation or demand-side resources, policy-makers in restructured jurisdictions are constrained in the design problem because SOs must procure rather than provide balancing flexibility. In the absence of ownership restrictions, it might be more effective and/or efficient for the SO itself to provide balancing flexibility. Referring to ancillary services (of which frequency control services are a subset), Pollitt and Anaya (2019) argue that in the absence of limits on SO ownership, R. H. Coase’s insights on the nature of the firm suggest that transaction costs would determine whether the SO provides ancillary services themselves or outsources their provision via procurement through spot markets (markets that involve immediate trade) (Coase, 1937):

[Ancillary services are often] associated with significant uncertainties about how much to procure and may be subject to significant market power within the limited local area that they are needed (especially for voltage support and constraint management). (…) Internal production works well in conditions of uncertainty about how much to procure and/or how the costs of different quality features trade off with each other. In-house production can also be a good way to manage external suppliers who would otherwise exercise market power.

This excerpt alludes to the “quality constraint” imposed by the construct of (spot) markets. As I discuss in Section 4.4.3.2 and Section 5.3, spot markets work best with well-defined, fungible and discrete products. However, such products ignore interdependencies and the wide technical capability “spectrum” of power system resources (Gimon, 2020). The question here is whether the potential benefits of spot market competition and transparency outweigh those of SO coordination and a more nuanced or layered approach to the procurement of different services. Coordination is particularly desirable for balancing flexibility services that are “lumpy” and/or inseparable from other services. For example, mechanical inertia provision requires the commitment of synchronous generation, which in turn augments rotor angle stability and system strength (Billimoria et al., 2020). A practical (or “second-best”) solution to procuring these types of balancing flexibility is for the SO to use long-term bilateral contracts. Contracting can be a preferable procurement mechanism in the presence of uncertainty or market power, or where there is a requirement for a tailored service or a multi-faceted product (Pollitt and Anaya, 2019). If there are benefits to be gained from competitive pressure, contracts can be awarded to market participants through tender processes designed to assess a single criterion (e.g. price) or multiple criteria (e.g. price, community benefit and delivery risk) (Y. Rebours et al., 2007).

4.5.2.4 The design problem is underdetermined

The design problem for balancing practices is underdetermined. In other words, there are many possible design solution configurations. As I argue in Section 5.3, this is because some processes, services and markets have overlapping roles and functions. The primary challenges of a wide solution space are that policy-makers must determine the trade-offs between different design objectives for each solution configuration (e.g. between complexity, reducing constraints on the system and the level of nuance or layering in the suite of power system services), and must also assess which trade-offs are most acceptable across all practical solution configurations (van der Veen and Hakvoort, 2016).

4.5.2.5 Grid architectures

Power systems are becoming increasingly distributed as they integrate large numbers of consumer-owned energy resources. This shift has prompted policy-makers to propose new grid architectures that re-envisage the roles of and relationships between actors and resources in their jurisdictions (Conejo and Sioshansi, 2018). Examples include architectures that build power system resilience through interconnected microgrids (Hanna and Marqusee, 2022), or those that enable consumer-owned energy resources to actively participate in transmission or distribution-level real-time markets (Kristov et al., 2016; Schittekatte and Pototschnig, 2022). Transitioning to any one grid architecture requires forward-looking balancing practice design; however, given that long-term planning involves deep uncertainties (Lambert et al., 2003), policy-makers must maximise design flexibility (or alternatively, reduce the number of design and system constraints imposed by operational practices) to retain option value (Fisher, 2000).

4.5.2.6 Diversity of initial conditions and outcomes

The design process has proceeded differently across jurisdictions due, in part, to technological, infrastructural, institutional, geographical and behavioural differences (Ahlqvist et al., 2022; Papaefthymiou et al., 2018). These differences are likely to persist as jurisdictions revisit their balancing practices due to the path dependency of energy systems (i.e. initial “lock-ins” have a large bearing on what system trajectories and futures are considered to be possible) (Fouquet, 2016). Diversity of initial conditions and outcomes poses a challenge to the design problem because policy-makers in one jurisdiction must judge the experiences with a particular practice in another jurisdiction within that context. In other words, experiences from one jurisdiction do not transfer straightforwardly to another.

4.6 Conclusion: the knowledge gap

In light of increasing penetrations of VRE, policy-makers worldwide are contemplating redesigning operational balancing practices in their jurisdictions. The task of designing effective and efficient operational practices requires them to build an understanding of several facets of the design problem, including:

1. The technical capabilities and constraints of power system resources and system control strategies;
2. The socioeconomic objectives of the power system;
3. The institutional arrangements of a jurisdiction; and
4. The broader cultural and political factors that have influenced the organisation of the electricity industry.

Though developing this understanding is a significant undertaking in and of itself, the design problem is made even more complex by 1) the presence of existing tensions and challenges and 2) a changing resource mix. The first means that whilst “operating power systems primarily relying on [VRE] is technologically possible”, the design process is “institutionally complex” (Papaefthymiou and Dragoon, 2016, p. 81). The second invites new perspectives on challenges and solutions – for example, proponents of stronger price formation in real-time markets now not only argue that it is essential for efficient operation of and investment in utility-scale resources, but also that it is necessary so that the market can serve as a means to coordinate distributed consumer-owned energy resources (Hogan, 2019).

Policy-makers must confront and account for these complexities to design the flexible operational practices required to effectively and efficiently operate complex power systems and electricity markets undergoing a transition to renewable energy. Whilst the literature has identified high-level design outcomes and the design areas that deserve the most attention, there is, as Mays (2021) indicates, a role for empirical work that identifies “second-best” design solutions given the specific context of each power system and jurisdiction. This approach to the design problem recognises that purpose-fit balancing practices arise not from “optimal” settings, but from solutions that combine and compromise.

With the need for robust design decisions supported by context-specific analysis, the work in Chapters 4, 5 and 6 contributes to the field of power system operational practice design by providing recommendations to system operators, market designers and energy system policy-makers based on models and detailed empirical studies of facets of the Australian National Electricity Market. Whilst some of the recommendations in these chapters may be unsuitable for other jurisdictions, the work in this thesis also contributes to the broader literature by providing a framework for policy-makers elsewhere of how to approach “second-best” and context-specific design when assessing which practices are best to successfully balance electricity markets in operational timeframes with growing penetrations of variable renewable energy.

5 Research framework

This chapter outlines the motivating research question of this thesis, the objectives within the scope of this thesis and the research methods used to achieve these objectives. The research framework of this thesis is shown in Figure 7.

5.1 Research question and objectives

The research question that this thesis aims to address is outlined below:

Given existing design challenges and increasing penetrations of variable renewable energy resources as energy transition proceeds, how should we design operational practices for balancing electricity markets?

As the full scope of this research question is not possible to address within a single doctoral thesis, I focus on aspects of the design problem that are encompassed by the three following research objectives.

5.1.1 Research Objective 1

To determine what features are needed in centrally-coordinated arrangements for procuring frequency control services during energy transition.

Frequency control services are a significant component of a system operator’s balancing toolkit and are critical to ensuring that imbalances are quickly addressed. The design of frequency control arrangements in restructured electricity industries will need to be revisited as resource mixes, network topologies and broader market arrangements and policy settings change with growing penetrations of variable renewable energy resources. Existing literature on the design of frequency control arrangements has tended to focus on arrangements in a particular jurisdiction with limited consideration of international experience from jurisdictions with alternative arrangements. Furthermore, few existing studies consider the design problem through both an engineering and economics lens, and even fewer focus on what features are required to ensure that frequency control arrangements can meet the demands of energy transition for years to come. It is also worth noting that previous studies of the design of frequency control arrangements in the NEM do not capture recent experiences with deteriorating frequency performance, the reintroduction of mandatory requirements and the integration of high shares of VRE.

Chapter 4 aims to address these shortcomings in the existing literature and provide a perspective on what features are desirable in the market-based frequency control arrangements as jurisdictions decarbonise. In this chapter, I provide a high-level overview and comparison of the key features of frequency control arrangements in North America and Central and Western Europe, and identify the most prominent challenges to designing effective and efficient frequency control arrangements and potential solutions to these challenges. I then assess the performance of evolving arrangements in the NEM in delivering improved frequency control outcomes, with particular regard to growing renewable penetrations and evident tensions between mandatory requirements and market-based incentives. Based on this assessment and a discussion of the trade-offs between effective and efficient outcomes, I draw out four key insights on designing frequency control arrangements as power system capabilities and needs change: 1) Understanding control action interactions, 2) implementing efficient price formation and cost-allocation mechanisms, 3) monitoring and assessing service provision to better align participant remuneration with service quality, and 4) considering both regulatory and market mechanisms and their consequences and interactions.

5.1.2 Research Objective 2

To better understand how balancing flexibility capabilities in scheduling timeframes are changing during energy transition, and how these changes might impact the suitability and design of more decentralised operational balancing practices.

In wholesale electricity markets, market participation decisions determine the type and quantity of balancing flexibility available within scheduling timeframes. There is a role for empirical studies examining whether decentralised operational balancing practices, such as markets for services and products, are purpose-fit to deliver the balancing flexibility requirements of electricity markets in transition – particularly given that new markets for flexibility (e.g. reserve product markets) can introduce additional costs, constraints and complexity, and even encroach upon the functions of existing operational practices. Previous studies have neglected this consideration and have, for the most part, proposed or relied upon flexibility quantification methods that are less useful to policy-makers assessing, comparing and designing changes to decentralised balancing practices. Furthermore, there has been limited work that examines how the duration of certain types of balancing flexibility might affect the suitability of certain decentralised operational balancing practices.

Chapter 5 addresses this gap in the literature by quantifying balancing flexibility capabilities in scheduling timeframes both now and into the near future to inform market design. In this chapter, I offer a practical method for quantifying the time-varying spectrum of upwards and downwards balancing flexibility capabilities, and then use this method to assess historical and projected resource mixes in two regions of the Australian NEM. The results from my analysis suggest that with higher penetrations of renewable energy: 1) downwards flexibility margins can be exhausted around noon if wind and solar are unable or unwilling to provide it, 2) upwards flexibility becomes more scarce during morning and evening peak demand events and 3) a greater portion of upwards flexibility is provided by energy-limited resources. Using these findings, I examine and compare the suitability of various flexibility design options, with a particular focus on assessing the need for an additional reserve product market. I recommend that policy-makers examine how existing operational practices can be augmented to elicit upwards flexibility provision, and that duration specifications and sustained footroom procurement be considered for reserve products.

5.1.3 Research Objective 3

To explore how more decentralised operational balancing practices can be configured to maximise the deployability of balancing flexibility in scheduling timeframes.

Understanding the quantity of balancing flexibility available in a power system is inadequate if market participants are unable or unwilling to offer resource flexibility into wholesale electricity markets. Market participation decisions and thus resource schedules are informed by knowledge processes, which provide current and forecasted power system and market information. As such, these knowledge processes and, more broadly, market participation rules must be purpose-fit to enable resource scheduling that leads to effective and efficient system balancing. To the best of my knowledge, there are only a few studies that discuss the significance of market information and market participation rules in scheduling power system resources in electricity markets. However, none of these studies use empirical evidence to support scheduling or policy recommendations. Furthermore, whilst previous studies have examined the impact of imperfect foresight on energy storage resources with different storage durations, they do not simulate storage operation in fast markets (i.e. dispatch decisions every 5 minutes) with high real-time prices and significant penetrations of VRE.

Chapter 6 explores how market information and market participant operational strategies impact the deployability of balancing flexibility from energy-limited storage resources, which are expected to aid in balancing electricity markets with high penetrations of variable renewable energy through energy arbitrage. In this chapter, I focus on the scheduling coordination role of centralised price forecasts generated by the system and market operator in Australian NEM. I highlight the increasing frequency and severity of errors in these price forecasts, and propose a hypothesis that market participant (re)bidding is partially responsible for this phenomenon. I then model the extent to which arbitrage revenues might be reduced (compared to perfect foresight operation) should these forecasts guide battery energy storage scheduling. Based on the findings from these analyses, I discuss potential changes to market participant scheduling strategies and market design that could improve scheduling outcomes. I recommend that Australian policy-makers not only increase the frequency at which centralised knowledge processes are run, but also consider whether stricter market participation restrictions might incentivise participant bidding strategies that are less likely to induce sudden price forecast swings that can hamper effective scheduling.

5.1.4 Case studies of the Australian National Electricity Market

Whilst this thesis aims to achieve these research objectives and thus produce insights for policy-makers worldwide, I draw on experiences from the Australian NEM and use its resource configurations and market arrangements in the case studies contained within this thesis. Research from the NEM should be of interest to electricity market designers elsewhere for several reasons:

1. Relative to other jurisdictions, the NEM regularly experiences high instantaneous penetrations of renewable energy (a maximum of 72.9% in November 2023⁸) and variable renewable energy (a maximum of 71.9% in November 2023⁹). As such, the changes in balancing flexibility capabilities and requirements in the NEM that are discussed in this thesis foreshadow those that are likely to occur in other jurisdictions as they decarbonise their electricity supply.
2. Similarly, the NEM has seen world-leading deployments of rooftop solar PV and other consumer-owned energy resources. These resources constitute a significant part of the resource mix (as high as 48% in a single market interval in October 2023¹⁰). Electricity market designers worldwide are becoming increasingly concerned with the impact of these resources on system balancing and are considering how best to facilitate the provision of balancing flexibility from these resources to the benefit of both the system and consumers. In this thesis, I touch on the impact of consumer-owned energy resources on balancing, and which operational features and practices best enable these resources to offer balancing flexibility to the wider system. However, distributed and consumer-owned energy resources are not a focus of this thesis.
3. Many of the world’s electricity markets are considering or adopting design features, such as short market intervals and faster frequency control service markets, that have long been a part of the NEM’s design. As such, experience from the NEM documented in this thesis can inform how or even whether certain design features should be implemented elsewhere.

5.2 Research methods

To achieve the objectives outlined in the previous subsection, I use three research methods: literature review, system and market data analysis and techno-economic modelling. In the subsections below, I provide an overview of these methods. I describe the methods used to address each research objective in detail in the methodology sections of Chapters 4, 5 and 6.

5.2.1 Literature review

Academic and industry literature from the NEM and other jurisdictions on the design of operational balancing practices was reviewed to better understand the nature of the design problem, and the success of and remaining challenges with the implementation of various balancing practices. Though a detailed literature review was completed for each study, a large portion of the output from the literature review is in Chapter 4.

5.2.2 System and market data analysis

Various analyses of system and market data from the Australian NEM were completed in Chapters 4, 5 and 6 to assess system and market outcomes, and to provide empirical evidence for market modelling assumptions (next subsection) and policy discussion in this thesis. Data was obtained using NEM-specific open-source packages including NEMOSIS (Gorman et al., 2018), NEMSEER (Prakash et al., 2023b), the AEMO Monthly Data Archive tool (Prakash, 2023b) and nem-bidding-dashboard (Gorman and Chambers, 2023), and analysed in Python and Julia (Bezanson et al., 2017) using pandas (The pandas development team, 2020) and DataFrames.jl (JuliaData, 2023) , respectively. Plots were generated using matplotlib (Hunter, 2007), plotly (Plotly Technologies Inc., 2015) or Makie.jl (Danisch and Krumbiegel, 2021).

5.2.3 Techno-economic (market) modelling

Techno-economic optimisation modelling was completed as a part of the work presented in Chapters 5 and 6. In Chapter 5, two market regions of the Australian NEM were modelled using PLEXOS, a commercial power system and electricity market modelling tool (Energy Exemplar, 2021). It enables scheduling processes and bid-based electricity markets to be modelled. The longer-term scheduling and market modelling problems formulated by PLEXOS were solved using the CPLEX solver (IBM, 2021). In Chapter 6, an open-source Julia optimisation package (JuMP) was used to formulate mixed-integer linear programs that were then solved to produce market participation schedules for energy storage resources with different storage durations and operational objectives (Lubin et al., 2023). Each of the optimisation problems was solved using the open-source HiGHs solver (Huangfu and Hall, 2018) on Katana, a high-performance computational cluster supported by UNSW Research Technology Services (UNSW Research Technology Services, 2023). Results from Chapter 5 were analysed using pandas (The pandas development team, 2020) and plotted using matplotlib (Hunter, 2007), and results from Chapter 6 were analysed using DataFrames.jl (JuliaData, 2023) and plotted using Makie.jl (Danisch and Krumbiegel, 2021).

6 Frequency control arrangements: insights from the National Electricity Market

6.1 Link to thesis

Frequency control services are a significant component of a system operator’s balancing toolkit and are critical to ensuring that imbalances are quickly addressed. The design of frequency control arrangements in restructured electricity industries will need to be revisited as resource mixes, network topologies and broader market arrangements and policy settings change with growing penetrations of variable renewable energy resources. This chapter aims to provide a perspective on what features are desirable in the market-based frequency control arrangements as jurisdictions decarbonise by drawing on experiences from the Australian National Electricity Market. The purpose of this chapter is to address the first research objective of this thesis:

To determine what features are needed in centrally-coordinated arrangements for procuring frequency control services during energy transition.

The following peer-reviewed journal article published in Renewable and Sustainable Energy Reviews constitutes this chapter:

Prakash, A., Bruce, A. & MacGill, I. Insights on designing effective and efficient frequency control arrangements from the Australian National Electricity Market. Renewable and Sustainable Energy Reviews 161, 112303 (2022).

6.2 Abstract

For restructured electricity industries undergoing energy transition, designing effective and efficient frequency control arrangements is a complex and ongoing task that requires appropriate configuration of controllers, generator technical connection requirements, market arrangements and wider policy settings. In this paper, we provide an overview and assessment of these arrangements in Australia’s National Electricity Market - a useful case study given its long-standing frequency control ancillary services markets, yet recent challenges in maintaining secure frequency control. We assess the performance of these evolving arrangements in delivering improved frequency control outcomes, with particular regard to growing renewable penetrations and evident tensions between mandatory requirements and market-based incentives. Based on this assessment, we draw out four key insights on designing frequency control arrangements as power system capabilities and needs change: 1) Understanding control action interactions, 2) Implementing efficient price formation and cost-allocation mechanisms, 3) Monitoring and assessing service provision to better align participant remuneration with service quality, and 4) Considering both regulatory and market mechanisms and their consequences and interactions. In particular, we discuss the trade-offs between effective and efficient outcomes, and provide arguments for more robust and forward-looking frequency control arrangements during energy transition.

6.3 Introduction

As a consequence of growing momentum to address global warming and continually declining technology costs, many power systems around the world are undergoing an energy transition in which significant capacity additions of variable renewable energy (VRE) and other inverter-based resources (IBR) are being accompanied by the progressive retirement of existing fossil fuel generation (International Energy Agency, 2021). Such power systems are currently experiencing or expected to soon experience high instantaneous penetrations of VRE (i.e. beyond 50% of grid demand being met by VRE at any given time), which can pose technical challenges to the stable and secure operation of a power system (Kenyon et al., 2020; Kroposki et al., 2017; Meegahapola et al., 2021). While several of these challenges have technological solutions of various maturities, configuring mechanisms in an effective and efficient manner across power system design layers, which span from how resources are controlled to how grid codes and markets are designed, remains an open and significant challenge.

In this article, we focus on one aspect of power system security: control of AC frequency. Maintaining frequency near the nominal value of a power system (either 50 or 60 Hz) is contingent on the ongoing balance of active power supply and demand within a synchronous area (Grainger, 1994). Power system frequency deviations are a consequence of instantaneous supply-demand imbalances, which typically occur as a result of system variability (predictable changes in supply or demand, such as fluctuations and ramps of generation or load) and uncertainty (unpredicted changes in supply or demand, such as forecast errors or unplanned outages) (Ela et al., 2011). System operators (SOs) achieve short-term active power balancing using reserve capacity. Whilst there are many names for these reserves¹¹, this article will focus on a common subset that responds to and mitigates frequency deviations over short timeframes (milliseconds to minutes). We will refer to such reserves as Frequency Control Services (FCS). If FCS are insufficient or inadequate, the system frequency may deviate beyond acceptable system limits and lead to equipment damage, load shedding, generator trips and cascading failures that lead to blackouts (Kirby et al., 2002; Ulbig et al., 2014).

In electricity industries with competitive markets for energy and FCS, frequency control arrangements consist of control, regulatory and market-based mechanisms (Mancarella and Billimoria, 2021). Control mechanisms specify the technical requirements for FCS. Regulatory and market-based mechanisms are used by the SO to:

1. Mandate or incentivise participant behaviour in the energy market that facilitates system balancing. This includes enforcing dispatch compliance or penalising participant portfolio imbalances; and

2. Procure FCS from capable resources (i.e. generators, loads and network elements).

Regulatory FCS procurement mechanisms are often mandatory and include equipment standards, connection requirements and SO intervention, whereas market-based FCS procurement mechanisms are often voluntary and include remunerative schemes and contract or spot markets. Together, these mechanisms dictate the physical effectiveness and productive, dynamic, price formation and cost-allocation efficiencies of FCS provision and procurement. Well-designed arrangements should be effective and efficient, where effectiveness entails sufficient and robust frequency response to meet physical power system requirements and efficiency relates to frequency response being provided at low cost, both now and into the future (Y. Rebours et al., 2007; van der Veen and Hakvoort, 2016).

As power systems transition towards higher instantaneous penetrations of VRE and IBR, SOs are likely to face the following challenges to short-term system balancing that may require existing frequency control arrangements to be revisited:

• VRE adds variability and uncertainty to a power system, particularly if similar technologies are situated within close proximity of one another (i.e. correlated production and/or forecast errors) (Australian Energy Market Operator, 2020d; Keeratimahat et al., 2021). Furthermore, unless an appropriate response is incorporated and enabled in their control systems, VRE and other IBR do not provide FCS. In jurisdictions that do not require, incentivise or allow VRE and IBR to provide FCS, the displacement of synchronous machines in dispatch has led to lower availabilities of resources that provide FCS (Australian Energy Market Operator, 2020g; Denholm et al., 2020; Milano et al., 2018) .

• In jurisdictions with competitive markets for energy and FCS, there is a tension between achieving economically efficient markets and the redundancy, certainty and control afforded to the SO. While the societal and economic costs of power system failure are often very large, it may be difficult for the SO to justify the cost of mitigation measures when they are ongoing or significant and when the joint probability of events or failures is low. The uncertainties associated with energy transition and the impacts of global warming are likely to present additional challenges. Power system security measures may need to be implemented rapidly and be both robust to a range of futures and resilient in the face of shocks, such as severe weather events (Eggleston et al., 2021; Prakash et al., 2021).

In this paper, we provide insights and recommendations on designing more effective and efficient frequency control arrangements based on experience from the Australian National Electricity Market (NEM). The NEM is currently experiencing relatively high system-wide instantaneous VRE penetrations (just over 60% in 2021) and is expected to experience penetrations as high as 75-100% by 2025 (Australian Energy Market Operator, 2021b, 2021c). Though the NEM’s frequency control arrangements were once arguably world-leading (Riesz et al., 2015; Thorncraft and Outhred, 2007), the speed at which system capabilities and needs are changing and the removal of mandatory requirements in 2001 as a part of a paradigm shift from obligation to remuneration for FCS have exposed design issues. In attempting to address these issues, the NEM’s rule makers have placed FCS obligations on generators and transmission network operators and have undertaken reforms to the NEM’s energy and FCS markets, including introducing a new market to procure emergency fast frequency response (FFR) from IBR. Whilst the NEM is an electrically-isolated power system with a relatively simple energy-only market, the insights and recommendations from this paper are likely to be relevant to other power systems and interconnections as their existing conventional generation retires and VRE deployment levels increase.

This paper offers three contributions to the literature. First, we provide a high-level overview and comparison of the key features of frequency control arrangements in North America and Central and Western Europe, and provide a review of the most prominent challenges to designing effective and efficient frequency control arrangements and the potential solutions discussed in the literature. Second, we provide a comprehensive update to previous literature on frequency control in the NEM (Riesz et al., 2015; Thorncraft et al., 2008; Thorncraft and Outhred, 2007). Our analysis benefits from recent experience in the NEM that encompasses deteriorating frequency performance, the reintroduction of mandatory requirements and integrating higher shares of VRE. While several of these aspects have been discussed independently in the literature, this paper seeks to provide a structured and holistic analysis of developments in the NEM and their implications for frequency control arrangement design. Third, this article advocates for designers placing a greater emphasis on delivering forward-looking frequency control arrangements during energy transition through the implementation of more robust regulatory mechanisms and ensuring that market-based mechanisms are capable of supporting FCS investment. As highlighted in the following sections, these design features have received surprisingly little attention in the literature.

The rest of the chapter is structured as follows. In Section 4.4, we provide an overview of typical frequency control arrangements, with a focus on restructured electricity industries in North America and Europe, and the main challenges faced in their design. We describe the NEM, its frequency control arrangements and the specific challenges posed by increasing penetrations of VRE and other IBR in Section 4.5. In Section 4.6, we analyse the performance of the NEM’s frequency control arrangements in responding to the challenges explored in Section 4.4, with primary frequency response and regulation (secondary frequency response) services in the NEM as case studies. Based on our analysis, we conclude by offering four key insights to operators, regulators and market-bodies that include understanding control action interactions; ensuring that arrangements are capable of supporting investment in FCS capability; monitoring, assessing and remunerating FCS performance; and considering both regulatory and market-based mechanisms in the design of effective and efficient frequency control arrangements.

6.4 Context

6.4.1 Conventional frequency control schemes

SOs employ hierarchical and sequential frequency control schemes. In most power systems, such schemes implicitly include inertial response and explicitly define FCS such as primary frequency response (PFR), secondary frequency response (SFR) and tertiary frequency response (TFR). In general, once frequency has deviated from the system nominal value, synchronous machines provide an inertial response that is inherent and immediate in slowing the rate of change of frequency (RoCoF). Within seconds, generators and/or loads provide autonomous and decentralised control action through PFR (Eto et al., 2018; Machowski et al., 2020). PFR arrests the frequency deviation to enable the slower and more centralised control actions of SFR and TFR to return the power system frequency to its nominal value (Ela et al., 2012b; Eto et al., 2010). Should system frequency continue to rise or fall beyond the system’s allowable limits, emergency protection schemes such as under-frequency load shedding (UFLS) and over-frequency generation shedding (OFGS) relays may be triggered. In some systems, RoCoF relays are also used to prevent high RoCoFs from tripping or damaging equipment and to contain frequency nadirs and zeniths (Akram et al., 2020; DGA Consulting, 2016; Miller et al., 2017a).

6.4.2 Procurement of frequency control services

Except for inertial response from synchronous machines, the SO procures FCS capacity from capable resources within its control area and, in the case of SFR and TFR, activates FCS energy if necessary. In electricity industries where the SO owns most if not all the generation assets (i.e. a vertically-integrated utility), the SO is able to jointly schedule generation and FCS capacity with knowledge of the condition of the system and the status and cost structures of their plant. However, many electricity industries have undergone some degree of restructuring, which has created a greater role for competitively-oriented decentralised decision-making (van der Veen and Hakvoort, 2016) . The diverse outcomes of restructuring processes and differences in technical characteristics (e.g. capabilities of resource mix and network topology) have led to a wide range of frequency control arrangements across power systems (Poplavskaya and de Vries, 2019; Y. Rebours et al., 2007) , which have been reviewed and compared extensively within industry and academic literature (Banshwar et al., 2018; Brooks and Lesieutre, 2019; Ela and Hytowitz, 2019; Hewicker et al., 2020; Lopez et al., 2020; Ocker et al., 2016; Y. G. Rebours et al., 2007a, 2007b; Reishus Consulting LLC, 2017; Zhou et al., 2016).

In restructured electricity industries, the provision of more passive FCS (e.g. ride-through capabilities) is usually mandated by regulatory mechanisms such as connection agreements and grid codes, whereas FCS that require additional response capabilities or impose opportunity-costs on suppliers are procured and remunerated by the SO through market-based mechanisms. In Section 4.4.2.1 & Section 4.4.2.2, we provide an overview of typical features¹² and key developments in market-based mechanisms for procuring FCS in North America and Central and Western Europe, respectively. These regions best represent the two prevailing short-term wholesale electricity market models: central dispatch markets, in which the SO issues dispatch instructions, and decentralised or self-dispatch markets, in which resource dispatch is managed by market participants (Ahlqvist et al., 2018). Given that FCS and energy are partially substitutable goods, the characteristics of short-term wholesale electricity markets heavily influence the design of FCS arrangements and thus these regions provide an interesting contrast. However, despite their differences, the SO plays a central role in both of these regions as they determine the area demand for FCS capacity, activate FCS energy as required and are ultimately responsible for ensuring that the power system is balanced and securely operated.

6.4.2.1 North American markets

In North America, central dispatch wholesale electricity markets are operated by an Independent System Operator (ISO) or Regional Transmission Organization (RTO) and are distributed across three synchronous areas. These markets consist of two short-term centralised platforms: a day-ahead market and a real-time market. In the day-ahead market, the SO solves a security-constrained unit commitment problem using supply offers (single or three-part) and demand bids (quantity or price-quantity) to produce day-ahead locational marginal prices and a financially-binding hourly schedule. In the real-time market, the SO solves a security-constrained economic dispatch problem (typically every five minutes) using generator price-quantity offers and a demand forecast to produce real-time locational marginal prices and a set of physically and financially binding dispatch instructions. Thus, each short-term market is cleared to maximise social welfare whilst respecting network and system security constraints (Chow et al., 2005; Cramton, 2017).

Except for Frequency Responsive Reserves (i.e. PFR), operating reserves (i.e. FCS capacity) are explicitly procured by placing an obligation on load-serving entities to self-provide or purchase their share from SO-run FCS markets (Ela et al., 2012b; Zhou et al., 2016). These FCS markets are usually integrated into day-ahead market and, in most jurisdictions, the real-time market. Standard products in North American markets include Regulation (i.e. SFR during normal operation), Spinning and Non-Spinning Reserves (i.e. TFR deployed following an event) (Ela and Hytowitz, 2019; Hewicker et al., 2020; Zhou et al., 2016). Participants can submit offers for FCS in addition to offer for energy. Unit commitment and economic dispatch permit co-optimisation of energy and FCS procurement. From the perspective of the SO, co-optimisation ensures that the total system cost of achieving an energy supply-demand balance is minimised alongside FCS requirements, subject to network and system security constraints. From the perspective of participants, co-optimisation leads to an FCS price that not only reflects the price offer of the marginal resource, but also any "profit" it forgoes in the energy market (assuming supplier offers reflect their short-run marginal costs) (Ela et al., 2012a; Isemonger, 2009). As such, ISO/RTO FCS markets can compensate opportunity-costs related to the day-ahead and/or real-time market but only allocate costs to load-serving entities through a procurement obligation.

Though North American FCS markets have predominantly procured and remunerated FCS capacity, ISO/RTOs (except Texas’ ISO, ERCOT) were ordered to also remunerate Regulation providers for the quantity of energy provided whilst accurately following control signals by the Federal Energy Regulatory Commission’s (FERC) Order 755 (Commission, 2011). As such, Regulation providers offer a quantity of capacity, a price for capacity and a price for "mileage", which is the energy delivered. Remuneration for Regulation takes performance (the ability of a resource to follow the ISO/RTO’s control signals) into account, though how this is implemented varies between ISO/RTOs (Ela and Hytowitz, 2019; Fernández-Muñoz et al., 2020). A notable example is the PJM RTO, which uses both a standard SFR control signal (RegA) and faster SFR control signal (RegD) intended for battery energy storage systems (BESS). PJM determines how interchangeable a resource’s RegD provision is with RegA provision (the marginal benefit factor) to clear the Regulation market and calculates a performance score for use in market clearing and settlement. However, according to the independent market monitor, the omission of the marginal benefit factor from market settlement has led to perverse market outcomes (Brooks and Lesieutre, 2019; Monitoring Analytics, 2021).

6.4.2.2 European markets

Most of the electricity markets of Central and Western Europe are self-dispatch and consist of two short-term platforms: the day-ahead market and the intraday market, which can be continuous, composed of frequently-run discrete auctions or a combination of the two. Each of these platforms is coupled across the majority of market zones in Europe, with a single price coupling algorithm used to simultaneously clear zonal day-ahead markets and a single order book compiled to match cross-zonal intraday orders (EPEX Spot, n.d.; NEMO Committee, n.d.). In contrast to North American electricity markets, the market operator is responsible for market operation and is distinct from the Transmission System Operator (TSO). Generation and load are managed by Balancing Responsible Parties (BRP), which must submit binding operational schedules to the TSO ahead of delivery (often by the day prior to delivery). As BRPs become aware of potential deviations closer to real time (e.g. improved forecasts), they are able to adjust their submitted schedules (i.e. remain "balanced") through trades on the intraday market (Lago et al., 2021b; Müsgens et al., 2014). BRPs face financial repercussions if they are imbalanced via an imbalance price and, in some jurisdictions, are legally obliged to be balanced (ENTSO-E WGAS, 2021).

Following gate-closure of the intraday market, residual imbalances are primarily addressed by FCS (known as balancing services) procured by the TSO. Standard FCS in Europe include Frequency Containment Reserve (i.e. PFR), automatic Frequency Restoration Reserves (i.e. SFR), and manual Frequency Restoration Reserves and Replacement Reserves (i.e. both TFR), with minimum technical requirements for each specified by the European Network of Transmission System Operators for Electricity (ENTSO-E) (European Network of Transmission System Operators for Electricity, 2013). Depending on the FCS product and the jurisdiction, TSOs may distinguish between FCS capacity (balancing capacity) and the delivery of FCS energy (balancing energy). The provision of one or both is mandated in some cases, but where both are procured competitively, Balancing Service Providers (BSP) typically submit separate offers for FCS capacity and FCS energy (Abbasy, 2012). FCS capacity markets are often cleared days to months in advance of real-time whereas the FCS energy market, which effectively constitutes merit-order or pro rata activation of capacity for FCS energy provision, is cleared within an hour or minutes of real-time (ENTSO-E WGAS, 2021; Ocker et al., 2016; Poplavskaya and de Vries, 2019). FCS capacity costs are typically allocated to power system users via a grid tariff. FCS energy costs are typically allocated to BRPs based on their schedule deviations and an imbalance price, which may differ from the FCS energy price paid to BSPs (Hirth and Ziegenhagen, 2015; Vandezande et al., 2010). As such, European FCS markets generally disincentivise causers of imbalance through the imbalance price, which may also recover or reflect the cost of FCS energy. However, since FCS capacity markets are typically decoupled from and cleared ahead of short-term energy markets, perceived opportunity-costs based on expected short-term energy market prices must be internalised within participants’ FCS offers.

Given the relatively high degree of interconnection between transmission systems in Central and Western Europe, cross-TSO initiatives are in place and being expanded to address imbalances and share FCS across the Continental Europe synchronous area. When sufficient cross-TSO transmission capacity is available, initiatives currently in place enable participating TSOs to jointly procure Frequency Containment Reserve capacity, net imbalances (i.e. reduce the demand for SFR by aggregating individual control area imbalances) and jointly procure automatic Frequency Restoration Reserve capacity and energy (European Network of Transmission System Operators for Electricity, 2020). Further efficiency gains are expected following the implementation of integrated market platforms for imbalance netting and balancing energy for SFR and TFR. The implementation of these platforms is mandated by the European Commission’s European Balancing Guideline and requires certain FCS product definitions and market features to be harmonised across the balancing energy markets of participating TSOs (50hz, 2017; European Commission, 2017).

6.4.3 Designing frequency control arrangements

As with any policy problem, designing frequency control arrangements in restructured electricity industries requires design principles, variables and performance criteria to be established. The public good characteristics of frequency control have heavily influenced arrangement design principles across jurisdictions, such as the common preference for the SO to centrally coordinate FCS procurement and activation (Müsgens et al., 2014; Y. Rebours et al., 2007). In contrast, though some design variables are common, others may only apply to particular systems based on their resource mix, network topology and/or market design. Y. Rebours et al. (2007) discuss design variables for central dispatch markets related to the following arrangement features:

1. FCS procurement;

2. Price formation, which when efficient should lead to FCS prices not only reflecting the true cost of the service, but also its true value to the system; and

3. Allocation of the cost of FCS.

Similarly, Abbasy (2012) discusses the main design variables applicable to European self-dispatch markets. van der Veen and Hakvoort (2016) build upon this work to provide a more comprehensive treatment of design variables in self-dispatch markets. Y. Rebours et al. (2007), Abbasy (2012) and van der Veen and Hakvoort (2016) all propose some variation of effectiveness and efficiency as performance criteria, with van der Veen and Hakvoort (2016) analysing the various trade-offs between and within each criterion.

Despite the well-defined nature of the design problem, there are several challenges to achieving effective and efficient arrangements. In Section 4.4.3.1 & Section 4.4.3.2, we present the most prominent challenges and their treatment in the literature.

6.4.3.1 The influx of VRE and other IBR in power systems

As discussed in Section 4.3, VRE adds variability and uncertainty to power systems which, at the very least, can lead to increased procurement and activation requirements for PFR and SFR during normal operating conditions (Ela et al., 2011). Three proposals to address this issue and thus reduce FCS requirements with growing penetrations of VRE have been discussed in the literature. The first is to shorten energy market trading/dispatch intervals (Ocker and Ehrhart, 2017; Riesz and Milligan, 2015) and the time between market gate closure and dispatch (Katz et al., 2019), thereby enabling scheduling based on up-to-date system conditions and forecasts. The second is to increase coordination between control areas within a synchronous area by netting imbalances (King et al., 2011), jointly procuring and dispatching FCS (Scherer et al., 2013) or aggregating them into a single market region (Milligan and Kirby, 2010; Riesz and Milligan, 2015). These two proposals alone have delivered significant system savings in Germany despite growing penetrations of VRE (Hirth and Ziegenhagen, 2015; Ocker and Ehrhart, 2017). The third is for the SO to determine the required quantity of FCS capacity (dimensioning) using dynamic and probabilistic approaches (as opposed to static and deterministic) that adequately reflect current or expected power system conditions and an acceptable level of risk, such as a reliability standard (De Vos et al., 2019; Holttinen et al., 2013; Ortega-Vazquez et al., 2020).

In recent years, SOs have become increasingly concerned with growing penetrations of asynchronous IBR leading to higher RoCoFs and fewer resources offering conventional FCS (Denholm et al., 2020; DGA Consulting, 2016; Hartmann et al., 2019). However, VRE and other IBR are able to provide tunable conventional FCS, FFR and/or an inherent response that strongly resembles the inertial response of synchronous machines¹³ if this is facilitated by arrangement design (Fernández-Muñoz et al., 2020; Mancarella and Billimoria, 2021; Miller et al., 2017b). Following a contingency event in a low-inertia power system, rapid FCS from IBR can mitigate higher RoCoFs, which when unabated can lead to deeper frequency nadirs and zeniths and the subsequent activation of UFLS or OFGS (Australian Energy Market Operator, 2017a; NERC Inverter-Based Resource Performance Task Force, 2020; Tielens and Van Hertem, 2016).

6.4.3.2 Achieving economic efficiency

Achieving short-run efficiency entails supplier costs being reflected in their offers and adequately propagated to FCS prices, and the SO assigning at least some portion of FCS costs to system users that create a need for procurement or activation. A widely used pricing approach in ISO/RTO co-optimised FCS markets is a marginal price which incorporates the marginal resource’s short-term market opportunity-costs and their offer, which could reflect potential mileage or wear-and-tear costs (Frew et al., 2021a; Zhou et al., 2016). Though improving cost-allocation has been repeatedly proposed in North American literature (Ela et al., 2012a; Isemonger, 2009; Milligan et al., 2011), FCS costs are predominantly socialised across loads based on demand or consumption. In Europe, however, much attention has been given to FCS market pricing, scoring (the order in which offers are selected) and cost-allocation. Specifically, literature on European FCS markets has explored whether pay-as-bid or uniform pricing better facilitates suppliers revealing their true costs (Hirth and Ziegenhagen, 2015; Müsgens et al., 2014; Ocker et al., 2018), the particular offers scoring should consider (Ehrhart and Ocker, 2021; Müsgens et al., 2014) and the design of imbalance prices to sufficiently incentivise short-term balancing (Hirth and Ziegenhagen, 2015; Papavasiliou, 2020; Vandezande et al., 2010). Regardless, both European and North American literature suggest that increased competition in FCS markets is a priority. This could be facilitated by enabling distributed and utility-scale VRE and IBR to qualify for FCS provision, reducing minimum offer quantities, separating raise and lower (positive and negative) products and increasing market clearing frequency and the time resolution of FCS products (Frew et al., 2021a; Hirth and Ziegenhagen, 2015; Lago et al., 2021b; Poplavskaya and de Vries, 2019). Despite the typically "shallow" nature of FCS markets (i.e. additional supply can significantly reduce prices (Riesz and Milligan, 2015)), dynamic efficiency has received considerably less attention. Notable exceptions include Papavasiliou (2020) and Frew et al. (2021a), who briefly discuss the potential for FCS scarcity pricing to better reflect the true value of system reliability and support investment in FCS.

An additional challenge in implementing efficient FCS markets involves the trade-offs that must be considered. As outlined in Section 4.3, some mechanisms that improve efficiency may come at the expense of visibility, control and redundancy afforded to the SO, which typically does not own any FCS-capable assets. The former is typically achieved using market-based mechanisms and the latter through regulatory mechanisms. Ela et al. (2012b), Billimoria et al. (2020), Mancarella and Billimoria (2021) and Lal et al. (2021) discuss several prerequisites for implementing market-based mechanisms and stress that balance between market-based and regulatory mechanisms may be required. However, achieving this balance can be challenging due to the asymmetry between the risk of an event and its consequences, and that between the benefits of market efficiency and the cost of resilient and robust mitigation measures (Lal et al., 2021; Mancarella and Billimoria, 2021). Another trade-off is the arbitrary definition of FCS products. Market-based mechanisms will work best when FCS are "discrete" commodities and fungible. However, this ignores the wide “spectrum” of resource technical capabilities. Favouring fungibility may obscure physical and control interdependencies between FCS and restrict or fail to incentivise higher quality provision, thereby leading to an inefficient overall outcome (Gimon, 2020; MacGill and Esplin, 2020).

6.5 Frequency control arrangements in the Australian National Electricity Market

6.5.1 Overview of the NEM

The NEM consists of five regions corresponding to the eastern and southern Australian states of New South Wales (NSW), Queensland (QLD), Victoria (VIC), South Australia (SA) and Tasmania (TAS) (Figure 8). In 2020, the NEM serviced a total electricity consumption of approximately 190 TWh/year and a peak demand of approximately 35 GW across a ‘stringy’ network over 5000 kilometres long with relatively weak interconnection between regions through interconnectors (Australian Energy Regulator, 2021; MacGill and Esplin, 2020). As high voltage DC transmission connects the island of Tasmania to the mainland state of Victoria, the NEM consists of two synchronous areas operated at a nominal frequency of 50 Hz: the mainland states and Tasmania. Due to the large distances involved, the NEM is not electrically connected to other markets.

The NEM is a single platform (real-time) energy-only market with no explicit capacity mechanisms. Unit commitment is managed by market participants, who must submit resource-specific offers for energy and Frequency Control Ancillary Services (FCAS) capacity in price-quantity pairs the day before delivery. These offers are subsequently used in a pre-dispatch process, which provides forecasted market information (e.g. generation and demand, interconnector flows, prices, etc.) to market participants. While prices in submitted offers are fixed, participants may change the energy volumes in their offer up to a few minutes before the delivery dispatch interval commences. As the NEM is single-sided, security-constrained economic dispatch is run every five minute to meet forecast demand at least cost, subject to network and security constraints. Much like ISO/RTO markets, energy and FCAS markets are co-optimised with respect to technical feasibility and cost (Australian Energy Market Operator, 2021d, 2017b). Real-time dispatch produces zonal marginal prices for energy and FCAS, which form the basis for market settlement in each of the NEM’s regions.

6.5.2 FCAS markets

The NEM’s competitive FCAS markets consist of eight separate raise and lower FCAS products that can be classed as regulation FCAS or contingency FCAS, with the former responsible for control when frequency is within the normal operating frequency band (NOFB) and the latter for when frequency deviates outside the NOFB after an event (see Table ¿tbl:fcs-fcas?). This is similar to arrangements in many ISO/RTO markets, where FCS are divided into event and non-event reserves (Ela et al., 2011).

Security-constrained economic dispatch includes system-wide and regional FCAS requirement constraints. Regulation and contingency FCAS are typically procured for and from all regions of the NEM in the absence of binding local constraints. Local requirements for FCAS procurement apply to Tasmania and to the other regions of the NEM if they experience network constraints, are at risk of separation or when islanded¹⁴ (Australian Energy Market Operator, 2015a, 2010). Prices are calculated for each region of the NEM based on the sum of the shadow prices of local and system-wide constraints and FCAS costs are allocated to market participants based on a "Causer Pays" principle, which bears similarities to imbalance penalties in European markets (Australian Energy Market Operator, 2015b). FCAS providers are paid for enablement (capacity provision) regardless of whether their capacity is activated (Australian Energy Market Operator, 2015b; Riesz et al., 2015; Thorncraft and Outhred, 2007).

For a resource to provide FCAS, it must meet pre-qualification criteria and undergo a registration process. Historically, FCAS was provided by thermal generation (predominantly coal and some gas), hydropower generation and some large loads, such as hydropower pumps and an aluminium smelter, as only resources associated with wholesale energy market participants were permitted to offer FCAS. In 2017, the first battery energy storage system (BESS) in the NEM began to offer FCAS and market reform enabled demand response (DR) aggregators to offer contingency FCAS without participating in the energy market (Aurecon, 2019; Australian Energy Market Commission, 2016). In recent years, new FCAS market entrants have included several DR aggregators, new BESS, distributed PV-battery virtual power plants and wind farms (the latter two through trials) (Aurecon, 2019; Australian Energy Market Operator, 2021e; Australian Energy Regulator, 2021). However, these new entrants tend to offer smaller volumes and there are still relatively few FCAS providers in the NEM, with no single FCAS product having more than 30 providers across the system or 8 providers in any one region (Australian Energy Regulator, 2021).

6.5.3 NEM operation and governance

The Australian Energy Market Operator (AEMO) is responsible for the operation of the market and power system in the NEM in accordance with the National Electricity Rules (NER). They act as a single buyer of dynamically-determined volumes of FCS. The Australian Energy Market Commission (AEMC) is responsible for making or amending rules for the NEM. Both AEMO and the AEMC provide operational and strategic advice to the Energy Security Board (ESB), which is responsible for coordinating market oversight and longer-term reform such as the ongoing post-2025 NEM market design framework. As the market regulator, the Australian Energy Regulator (AER) monitors compliance with and enforces the NER.

6.5.4 Challenges to frequency control posed by VRE and IBR

The rapid pace at which IBR have entered the NEM was preceded by the exit of FCAS-capable synchronous generation (Figure 9). Many of these IBR do not currently offer FCAS or any meaningful frequency response to deviations other than the most extreme. Furthermore, though updated equipment standards require distributed IBR to ride-through and/or respond to certain frequency deviations, some inverter models have been found to be non-compliant and there is still a significant number of legacy systems in the NEM (Australian Energy Market Operator, 2021f; Stringer et al., 2020).

The challenges that VRE and other IBR pose to frequency control have been exacerbated by the NEM’s network topology. Limited interconnection between regions reduces the NEM’s cross-regional balancing capabilities and increases the likelihood of synchronous area separation following power system events, a consequence of which is that local requirements for FCAS may apply (Australian Energy Market Operator, 2019b). Furthermore, correlated variability and uncertainty can arise from intensive development of similar utility-scale VRE in areas with good wind or solar resources (as might occur in the Renewable Energy Zones identified by AEMO’s least-regrets transmission planning study (Australian Energy Market Operator, 2020h)). This is also an issue at the distribution level given the significant installed capacities of rooftop solar PV located within proximity of one another in suburban areas (Australian Energy Market Operator, 2021a).

6.5.5 Features of NEM frequency control arrangements

Below, we highlight some noteworthy features of the NEM’s frequency control arrangements that complement or contrast previous analyses in Thorncraft and Outhred (2007), Riesz et al. (2015) and Thorncraft et al. (2008).

6.5.5.1 Control mechanisms:

• There is no explicit TFR FCS in the NEM. Security-constrained economic dispatch is run every five minutes and is expected to relieve PFR and SFR and address supply-demand imbalances (Australian Energy Market Operator, 2020a).
• PFR from contingency FCAS is only required to respond to frequency deviations outside the NOFB (50 \pm 0.15 Hz). When FCAS markets were implemented in the NEM in 2001, mandatory PFR around a tight deadband of \pm 50 mHz was removed from the NER (Australian Energy Market Operator, 2019c). Since then and prior to 2020, there was no explicit procurement or requirement for tight-deadband PFR provision within the NOFB. The decline in the provision of tight-deadband PFR in the NEM is discussed further in Section 4.6.1.
• The mainland synchronous area is controlled as one balancing area by AEMO’s AGC (i.e. no tie-line biased SFR) despite limited interconnection between adjacent regions (Australian Energy Market Operator, 2018a). AGC control performance is discussed further in Section 4.6.2.

6.5.5.2 Market-based mechanisms:

• There are relatively few limits imposed on FCAS participation. FCAS can be provided by any technology through variable, switched or hybrid controllers (Australian Energy Market Operator, 2020i). Furthermore, regulation and contingency FCAS products are unbundled into raise and lower services, and contingency FCAS products are unbundled based on response time. All of these features improve the potential for participation and competition in FCAS markets, though market participants can and often are enabled to provide multiple FCAS.
• FCAS unbundling has enabled a ‘Causer Pays’ cost allocation framework. Raise contingency FCAS costs, which are incurred as insurance for the failure of a generator, are distributed amongst generators in proportion to their generation in the trading interval. Similarly, lower contingency FCAS costs are distributed amongst loads based on their consumption in a trading interval. A complex methodology is used to calculate monthly, portfolio-wide Causer Pays contribution factors (outlined in Australian Energy Market Operator (2018b) and summarised in Riesz et al. (2015)) that determine how regulation FCAS costs are allocated to market participants. We discuss the issues associated with this methodology in Section 4.6.2.
• The NEM co-optimises FCAS that respond within similar timeframes. In the absence of constraints, the volume of 5-minute delayed contingency FCAS procured is reduced by the volume of regulation FCAS enabled (Australian Energy Market Operator, 2010).

6.5.5.3 Regulatory mechanisms:

• Connecting utility-scale generators negotiate the frequency response capability of their plant between a minimum access standard and an automatic access standard, the latter guaranteeing network access to the applicant. A suite of generator standards for frequency response were added to the NER in October 2018 and apply to any newly-connecting generation. These standards include minimum frequency disturbance ride-through times, automatic generation output reduction following extreme over-frequency events and the capability to operate in a frequency response mode with a proportional response¹⁵ (Australian Energy Market Commission, 2018a).

• Transmission Network Service Providers (TNSPs) are required to address any inertia shortfalls identified by AEMO within the NEM region in which they build, maintain, plan and operate the transmission network. AEMO’s assessment considers whether an islanded region can be securely operated following a contingency event. Shortfalls can be reduced by special protection schemes (e.g. disconnection of load following interconnector trip) and the provision of FFR, but they must ultimately be met by providers of inertial response (Australian Energy Market Operator, 2020k, 2018c).

6.6 Insights from the National Electricity Market

In light of existing challenges and those posed by energy transition, effective and efficient frequency control arrangements should enable sufficient FCS to be procured across timeframes and strike the appropriate balance between efficiency and robustness. In the following sections, we review issues associated with two core elements of the NEM’s frequency control hierarchy (i.e. PFR and SFR), assess their physical and economic performance and outline reform underway. Drawing on developments in the NEM and our review of arrangements in North America and Europe, we then discuss the merits and flaws of regulatory and market-based mechanisms with respect to sufficiency and efficiency. We conclude by offering insights that could serve as design principles for jurisdictions revisiting their frequency control arrangements during energy transition.

6.6.1 Declining tight-deadband primary frequency response

When FCAS markets were implemented in 2001, mandatory tight-deadband PFR was superseded by two types of PFR: voluntary PFR within the NOFB and competitive procurement for PFR outside the NOFB in the form of contingency FCAS (Australian Energy Market Operator, 2019c).

As such, the NEM’s frequency control scheme deviated from what has been argued to be international best practice as it only explicitly specified and procured wide-deadband PFR (i.e. deadband of \pm 150 mHz) (Australian Energy Market Operator, 2019c). In contrast, ENTSO-E specifies that PFR providers have a deadband no greater than \pm 10-15 mHz depending on the control area (European Network of Transmission System Operators for Electricity, 2013) and FERC Order 842 mandates all newly-connecting generation in US interconnections to operate frequency-responsive control equipment with maximum deadbands of \pm 36 mHz (Federal Energy Regulatory Commission, 2018a).

In recent years in the NEM, the lack of an incentive or requirement for tight-deadband PFR and perceived disincentives to its provision (through Causer Pays contribution factors discussed further in Section 4.6.2) has led to many synchronous generators that once provided tight-deadband PFR to widen deadbands or install control systems that block or dampen PFR from the speed governor within the NOFB (Australian Energy Market Commission, 2020b). Furthermore, many VRE generators were deployed in the NEM and connected with inverter control systems that were unresponsive to any frequency deviations other than the most serious.

The extent to which tight-deadband PFR provision had declined in the NEM and the consequences of this became clear to AEMO following a major power system incident on the 25^th of August 2018 (Australian Energy Market Operator, 2019d). Prior to the event, the QLD region was exporting \sim 900 MW to the rest of the NEM. Around 13:11:41, lightning strikes at the QLD-NSW interconnector resulted in the QLD region being separated from the rest of the NEM with excess supply. The SA region was exporting \sim 200 MW prior to the event and following QLD’s separation, this increased by more than 200 MW in response to under-frequency. The sudden increase in active power flow triggered an emergency scheme that disconnected SA from the NSW-VIC synchronous area, resulting in local over-frequency.

There were diverse responses from various generators following the double separation event. While many synchronous generators provided some form of PFR though not enabled for FCAS, their response was withdrawn by their load controllers in several cases so that the unit could return to its dispatch target (e.g. green and pink lines in top frame of Figure 10). Wind and solar farms were either unresponsive, tripped due to protection settings in their inverters, or reduced their active power output in line with performance standards negotiated in their connection agreements (middle and bottom frames in Figure 10). AEMO attributed slow frequency recovery and under-frequency load shedding in NSW and VIC to insufficient PFR from generators and a lack of appropriate contingency FCAS within the islanded regions. Over 50% of fast and slow raise contingency FCAS needed in NSW-VIC was enabled in SA and QLD, whilst QLD had no lower FCAS enabled to respond to over-frequency¹⁶ (Australian Energy Market Operator, 2019d).

Prior to this incident, deteriorating control of frequency within the NOFB was of concern to AEMO and the AEMC, and trials and investigations were recommended to inform the design of an incentive for tight-deadband PFR provision (Australian Energy Market Commission, 2018b). However, this separation event demonstrated the “urgent need for regulatory changes to arrest the ongoing decline in frequency performance in the NEM” and to enhance “the resilience of the NEM to similar major disturbances”, with AEMO submitting a rule change proposal for all capable generators in the NEM to provide mandatory PFR with a maximum deadband of \pm 0.015 Hz (i.e. 10% of the NOFB) (Australian Energy Market Operator, 2019c).

This rule was initially incorporated into the NER in 2020 as a temporary arrangement through the addition of a “sunset” after three years to demonstrate the AEMC’s commitment to investigating incentives or market-based mechanisms for tight-deadband PFR (Australian Energy Market Commission, 2020b, 2020c). AEMO has specified PFR settings, including maximum droop and response time, but is unable to require generation to reserve headroom for PFR (Australian Energy Market Operator, 2020m).

6.6.2 Performance and efficiency issues of regulation services

For SFR provided by regulation FCAS within the NOFB to be effective, the dynamics of the system need to accommodate slower SFR control action and the centralised secondary controller (in the NEM, AEMO’s AGC) needs to be properly configured. Prior to the introduction of mandatory PFR in the NEM, AEMO observed no significant improvement in NOFB frequency stability despite several increases in the minimum volumes procured for regulation FCAS in 2019 (Australian Energy Market Operator, 2019c). This is likely due to:

• A lack of fast and decentralised tight-deadband PFR supporting slower SFR;
• Inappropriate control signals being calculated within the AGC due to the use of rate limiters to account for ramping constraints, signal filtering and generator controller models that do not accurately reflect a unit’s frequency response (DIgSILENT, 2017). The latter is the consequence of an absence of control coordination between market participants and AEMO; and
• Variable communication delays between individual unit controllers and AEMO’s AGC system, and disparate response times from generators.

Furthermore, the control of all mainland regions as one balancing area can be problematic in the event of separation. AGC control of regulation FCAS enabled in islanded regions may exacerbate local frequency deviations when responding to the AGC frequency reference. This was the case during the double separation event on the 25^th of August 2018, in which the AGC instructed raise regulation FCAS generators in QLD and SA to respond to under-frequency in the AGC frequency reference despite local over-frequency (Figure 11). Such incorrect control action can occur until AEMO is able to manually reconfigure the AGC to treat each island as a control area - a process which can take up to 15 minutes (Australian Energy Market Operator, 2019d) .

Over time, inefficiencies in regulation FCAS procurement and cost-allocation have also become apparent. Regulation FCAS procurement in the NEM is dynamic beyond a minimum volume, but the dynamic component is based on the system time error (Australian Energy Market Operator, 2015a). Time error control is largely unnecessary as modern clocks no longer rely on power system frequency to keep the time (Y. Rebours et al., 2007). Furthermore, whilst AEMO is required to control the NEM within certain time error limits, these have been relaxed in recent years (Australian Energy Market Commission Reliability Panel, 2017). Given that time error is no longer prioritised as a control objective, dynamic regulation FCAS procurement based on better measures of sustained frequency deviation (e.g. mean absolute error as suggested by Riesz et al. (2015)) and/or a modelled distribution of potential intra-dispatch ramp uncertainty may be more suitable.

Regulation FCAS costs are allocated to market participants based on their contribution factor, a calculation which represents the extent to which the participant has contributed to the need for regulation FCAS through a deviation from a dispatch trajectory. Though the calculation methodology assigns weights to a generator or load’s dispatch trajectory deviation based on the AGC regulation direction and mileage requirement every 4 seconds, the disincentive for dispatch deviation suffers from a disconnect to causation. This is because the contribution factors of a generator or load are averaged over a 5-minute dispatch interval, summed over a 28-day period and then within a market participant’s portfolio (Australian Energy Market Commission, 2018b; Australian Energy Market Operator, 2018b; Australian Energy Regulator, 2020).

Much like portfolio-based balancing in Europe, the aggregation of contribution factors enables a market participant to offset antagonistic deviations with assisting deviations (from the provision of tight-deadband PFR) across its resources and time. However, the complexity and opacity of the methodology and cost-allocation process has contributed to the withdrawal of tight-deadband PFR in the NEM. Several generators disabled governor response in the NOFB in the belief that dispatch adherence alone will minimise Causer Pays liabilities (DIgSILENT, 2017).

6.6.3 NEM assessment and outlook

Though the introduction of competitive FCAS markets in 2001 initially resulted in significantly lower FCAS prices in the NEM (Riesz et al., 2015; Thorncraft and Outhred, 2007), volume-weighted average FCAS prices, particularly those for raise regulation and contingency services, have increased relative to the volume-weighted average energy price since 2016 (Figure 12). Furthermore, the increases in minimum regulation FCAS volumes and reductions in assumed load relief in 2019 have raised the procured volumes of regulation and contingency FCAS, respectively. Together, these factors have contributed to higher NEM-wide FCAS costs (Australian Energy Market Operator, 2019e). While quarterly FCAS costs were less than 1% of quarterly total NEM costs in 2015, 50% of all quarters from 2017 to 2020 had FCAS costs that were between 1-2% of total NEM cost (Australian Energy Regulator, 2021).

Prior to the implementation of mandatory PFR, higher NEM FCAS costs were arguably not accompanied by an improvement in frequency control performance. Alongside deteriorating frequency control performance within the NOFB (Figure 13), AEMO has expressed a loss of confidence in the NEM’s resilience to complex power system events, such as the double separation incident on the 25^th of August 2018 (Australian Energy Market Operator, 2019c). These events are typically more severe than the ‘credible’ contingency events (i.e. N-1 contingency) that dictate the volume of contingency FCAS procured.

Since the implementation of the mandatory PFR, settings specified by AEMO have been applied to a majority of large synchronous generators (> 200MW) and some smaller synchronous generators. Despite the absence of requirements for maintaining headroom and/or footroom, preliminary analysis by AEMO¹⁷ suggests that mandatory PFR has delivered better control of frequency within the NOFB (see Figure 14) and reduced excursions beyond the NOFB (Australian Energy Market Operator, 2021a). As a result of this initial success and further technical advice provided by AEMO, the AEMC has indicated that it intends to retain mandatory PFR at a tight-deadband following the “sunset” of the initial rule (Australian Energy Market Commission, 2021a).

However, this initial success may be a result of the headroom maintained by these generators for risk management purposes (e.g. defending contract positions) and any headroom made available to the system through the displacement of more expensive synchronous capacity by VRE. Given that several large synchronous generators are expected to retire in the coming decades (Australian Energy Market Operator, 2020h), continuing to rely on this “free” headroom (and any available footroom) into the future may reduce the potential resilience benefits of widespread, tight-deadband PFR and place a greater burden on generators that do reserve headroom and hence respond. The AEMC is proposing to address this issue by paying resources that provide assisting tight-deadband PFR (“double-siding”) (Australian Energy Market Commission, 2021a).

Presently, several other operational and market changes are being considered or implemented with the goal of improving the effectiveness of arrangements in the NEM. AEMO is investigating the use of dispatch constraints to (Australian Energy Market Operator, 2021g):

• Procure contingency FCAS volumes based on system inertia;
• Apply regional contingency and regulation FCAS requirements; and
• To limit the amount of switched contingency FCAS procured. Switched FCAS has a number of limitations compared to governor-like control (Australian Energy Market Operator, 2020g).

These additional constraints will likely improve the effectiveness of frequency control arrangements but may lead to higher FCAS costs. In addition to these procurement changes, the AEMC has made a rule to introduce raise and lower contingency markets for FFR by October 2023, each with a likely response time of 1 second (Australian Energy Market Commission, 2021b; Australian Energy Market Operator, 2021h). Whilst AEMO has highlighted that potential stability issues and interconnector maloperation will need to be managed (e.g. through delivery caps or provision constraints) (Australian Energy Market Operator, 2021i), these FFR markets, along with the ESB’s proposals for short-term scheduling and/or procurement of inertial response (Energy Security Board, 2021a), will likely improve AEMO’s operational toolbox for managing a low-inertia NEM.

6.6.4 Reactive regulatory requirements

Despite a broad set of FCS markets, there is a high degree of reliance on regulatory mechanisms in the NEM. Performance standards and mandatory PFR enforced by connection requirements in the NEM have recently been aligned with international grid-codes (Roberts, 2018). As argued by TNSPs and AEMO during the mandatory PFR rule change process, near-universal widespread provision of frequency control should lead to relatively low costs for individual participants and be outweighed by greater visibility and certainty for AEMO alongside the system-wide benefits of improved physical frequency control performance (Australian Energy Market Operator, 2019c; Dillon, 2019; Hopwood, 2019).

Regulatory mechanisms are ideal for mandating basic FCS capabilities as a condition for access, which may reduce the need to procure more specialised FCS, or where FCS faces significant barriers to efficient price formation or unbundled procurement. The latter reasons are particularly pertinent in the NEM. Current FCAS prices do not appear to be incentivising FCS provision from the vast majority of VRE generators, which have business models centred around energy provision (Australian Energy Market Commission, 2021c; Meegahapola et al., 2021). Furthermore, procuring inertial response is challenging due to its inseparability from system strength provision and unit commitment costs (Billimoria et al., 2020). With respect to these challenges, regulatory mechanisms in the NEM have assisted in ensuring some level of frequency response from most power system resources (e.g. mandatory PFR) and improving the ability of AEMO and TNSPs to coordinate the procurement of essential but “lumpy” FCS (e.g. inertia shortfall mechanism).

While mechanisms such as mandatory PFR are likely to improve the robustness of frequency control arrangements, it may be difficult for other regulatory mechanisms to keep in step with changing physical performance requirements in systems rapidly facing higher penetrations of VRE and IBR. Regulatory mechanisms are often only updated after a number of years to reduce the burden placed on connecting resources. As such, they are slow to respond to changing capabilities and requirements. This delay often makes new standards and requirements reactive rather than proactive. For example, AEMO can only review utility-scale generator technical performance standards every 5 years (Australian Energy Market Commission, 2018a), a timeframe in which the solar PV capacity installed in the NEM has more than quadrupled (2015-2020) (Australian PV Institute, n.d.).

Additional concerns with regulatory mechanisms include poor dynamic efficiency and opaque costs(Riesz et al., 2015). In the absence of remuneration or incentives, particularly those that are linked to the quality of frequency response, there is no incentive to innovate or invest in higher-quality frequency control capabilities (Meegahapola et al., 2021). Furthermore, cost opacity may lead to FCS provision costs being internalised within other prices (e.g. energy) by participants and prevent the implementation of imbalance or dispatch non-conformance disincentives through cost-allocation mechanisms.

6.6.5 Preference for market-based arrangements

Since the establishment of the NEM, a competition norm has been established, with markets being viewed as a key driver for delivering the National Electricity Objective of “efficient investment in, and efficient operation and use of electricity services” (Haines and McConnell, 2016; MacGill et al., 2020a). This norm has pervaded all levels of participation and governance in the NEM. Generator owners opposed the mandatory nature of the mandatory PFR rule change on the basis that a lack of remuneration was against market principles and that it would lead to economically inefficient outcomes (Rolfe, 2019; Scott, 2019; Skinner, 2019). AEMO did not include a headroom requirement in its proposal, making the mandatory PFR rule change more palatable to market bodies and participants. The AEMC, who have expressed a clear preference for market-based approaches (Australian Energy Market Commission, 2018b), included a “sunset” clause in their initial decision to implement mandatory PFR. Furthermore, a market for FFR will be implemented in 2023 and the ESB’s post-2025 market design process is considering new system services markets for inertial response and TFR (Energy Security Board, 2021a, 2020b).

If incentives or remuneration are designed correctly, markets can drive short-run efficiency. Where required, they can also support investment in FCS capability and assist a power system in achieving dynamically efficient frequency control arrangements. However, in some cases, simply introducing new FCS markets may serve as ‘patchwork’ solutions to existing control deficiencies and market failures. These deficiencies and failures could be partially addressed by improving FCS cost allocation processes, verifying FCS performance and linking incentives to higher quality provision.

As discussed in [Section 4.4), efficient Causer Pays cost-allocation mechanisms in FCS markets could provide suitable disincentives for dispatch non-conformance or imbalances. In the NEM, the aggregation of regulation FCAS Causer Pays contribution factors over time and a portfolio has resulted in a blunt frequency performance market signal. The solution to this problem may not be as simple as strengthening disincentives (e.g. as proposed by Hirth and Ziegenhagen (2015) and Papavasiliou (2020)) for resource-based cost-allocation processes as potential exposure to high instantaneous FCS costs may lead to participants curtailing or decommitting flexible resources rather than providing an assisting frequency response. This has been observed in the NEM when local constraints have resulted in regulation FCAS (Australian Energy Market Commission, 2018b) and contingency FCAS (Australian Energy Market Operator, 2020n) price spikes. The AEMC has proposed a compromise to this problem by shortening the settlement period for regulation FCAS Causer Pays to 5 minutes but only allocating the costs of regulation FCAS capacity that is activated by AEMO (i.e. the cost of any unactivated capacity is socialised across power system users) (Australian Energy Market Commission, 2021a).

An alternative to Causer Pays is to allocate costs based on needs (‘User Pays’), such that connected equipment imposing RoCoF or frequency constraints pay for FCS. ‘Users’ of frequency control currently include synchronous machines and IBR that have not been configured to ride-through higher RoCoFs and greater frequency deviations. Following more extreme frequency deviations, the former may suffer equipment damage whereas both have the potential to trip (DGA Consulting, 2016; Miller et al., 2017a). A User Pays approach to cost-allocation could encourage resources to be more resilient to frequency deviations and thereby reduce system FCS costs (Lal et al., 2021), particularly if a significant proportion of connected equipment are IBR that can be configured to ride-through such disturbances.

Beyond choosing who costs should be allocated to and what an appropriate granularity for cost-allocation might be, market designers should ensure that the chosen methodology is transparent, can be understood by participants and that any calculations can be replicated using accessible data. If appropriate design choices are made, efficient cost-allocation could create counter-parties for financial instruments that hedge price risk (Skinner et al., 2020; Thorncraft and Outhred, 2007). FCS derivatives may drive investment in FCS capabilities by supporting business models in which FCS is a major revenue stream (this is currently the case for utility-scale BESS, DR aggregators and virtual power plants in the NEM) and assist in FCS price formation (Billimoria et al., 2020; Pollitt and Anaya, 2019).

As in ISO/RTO Regulation markets, aligning FCS procurement and/or remuneration with performance essentially recognises that there is a spectrum of FCS capabilities. This recognition is lacking in the NEM, where battery energy storage systems are responding precisely and rapidly to AGC regulation signals but are being paid the same as thermal plant that provide lower quality regulation FCAS (Australian Energy Market Operator, 2018d). However, implementing performance-based design is contingent on the SO verifying FCS provision. While AEMO has outlined FCAS delivery measurement standards and verification principles (Australian Energy Market Operator, 2020i), delivery verification appears to be restricted to confirming contingency FCAS delivery following a power system event (to the authors’ best knowledge). While a regular verification process does not appear to be in place for regulation FCAS, AEMO is proposing to specify minimum control requirements (e.g. response delay and ramp rate) and implement a regular testing cycle for resources registered for regulation FCAS (Australian Energy Market Operator, 2021j).

Market designers may also need to consider price formation in FCS markets to ensure that arrangements are at least capable of supporting investment during energy transition. As discussed by Hirth and Ziegenhagen (2015), VRE have low to no short-term energy market opportunity-costs when providing lower/negative FCS but can incur significant short-term energy market opportunity-costs when providing raise/positive FCS. The raise/positive opportunity-cost may be even higher if the SO requires additional curtailment to better ensure that FCS capacity is firm, which AEMO has required, or if the resource has entered into an energy off-take agreement, which is common in the NEM (Australian Energy Market Operator, 2018e). While co-optimised FCS markets mean that such opportunity-costs can be accounted for, FCS prices can be suppressed if large conventional generators with low to no opportunity-costs offer large volumes of FCS. Low prices can limit the incentive for high capital, low operating cost IBR to provide and invest in FCS capabilities. This may lead to a dynamically inefficient outcome as additional conventional generators are retired and limited FCS capabilities are offered by VRE and other IBR (Ela et al., 2019; Meegahapola et al., 2021). As discussed in Section 4.4.3.2, one potential solution to this issue is to strengthen scarcity pricing in FCS markets. The AEMC and ESB have discussed implementing system demand curves with scarcity pricing for all existing and proposed FCAS (Australian Energy Market Commission, 2020c; Energy Security Board, 2020b). However, the shape of these system demand curves and how they account for interdependent or interchangeable FCAS will ultimately dictate their success.

6.7 Conclusion

Whilst recent years have seen increasing participation from demand response and IBR, energy transition and a pervasive competition norm have exposed design issues in the NEM’s frequency control arrangements. As such, considerable attention and effort have been devoted to reforming the NEM’s arrangements in the past two years.

From our review of North American and European frequency control arrangements and our analysis of the NEM’s, we share four key insights below that could serve as design principles for operators, regulators and market-bodies attempting to design effective and efficient frequency control arrangements in restructured electricity industries during energy transition:

1. Control deficiencies may not be addressable through introducing new FCS. While this solution may address emerging needs (e.g. low-inertia operation), SOs and market bodies need to better understand the interdependency, interoperability and interchangeability between FCS and the interactions with other technical attributes of the power system (e.g. system strength) to ensure that frequency control is first and foremost effective. Once this has been achieved, the short-run efficiency of arrangements can be improved through mechanisms such as dynamic and probabilistic dimensioning and co-optimising the procurement of interchangeable FCS.

2. Given the pace and scale of energy transition, a dynamically efficient outcome in some power systems may require additional investments in FCS capability. FCS prices can be strengthened through scarcity pricing, which may better reflect the system’s preference for security and reliability. Such pricing mechanisms are complementary to appropriate and efficient cost-allocation based on causation or needs. Both efficient price formation and cost-allocation will improve the potential for FCS derivatives, which may assist in providing price signals for investment.

3. SOs should systematically and frequently verify FCS delivery, where relevant, and withhold or penalise remuneration when delivery is deemed to be insufficient. If such monitoring is in place, FCS remuneration can be performance-based to drive the provision of high quality FCS. Performance monitoring would also enable the SO to assess FCS arrangements and identify any deficiencies in control action or procurement.

4. During energy transition, a suitable set of frequency control arrangements will most likely involve a combination of market-based and regulatory mechanisms. Frequency control is a power system public good and achieving frequency stability requires a degree of coordination and cooperation between resources. These characteristics make it difficult to establish complete markets for FCS, and an emphasis on market solutions may obscure these characteristics to market participants and undermine effective control. In contrast, regulatory mechanisms may prove to be more robust and resilient in the face of uncertainties, particularly those that are exogenous to the power system (e.g. climate risk). Regardless of whether arrangements are skewed towards market-based mechanisms or regulatory mechanisms, designers should be more forward-looking and avoid assumptions regarding the provision of FCS capability over time, particularly when there is a pervasive competition norm and effective frequency control relies on sequential and hierarchical control actions.

7 Quantifying reserve capabilities: an Australian case study with increasing penetrations of renewables

7.1 Link to thesis

In wholesale electricity markets, market participation decisions determine the type and quantity of balancing flexibility available within scheduling timeframes. Empirical studies are required to examine whether decentralised operational balancing practices, such as markets, are purpose-fit to deliver the balancing flexibility requirements of electricity markets in transition. Assessing existing arrangements and the need for new markets for flexibility is critical as simply implementing the latter (e.g. a new reserve product market) can introduce additional costs, constraints and complexity and even encroach upon the functions of existing operational practices. This chapter focuses on understanding the balancing flexibility capabilities available in scheduling timeframes both now and into the near future, and using this knowledge to inform market design. The purpose of this chapter is to address the second research objective of this thesis:

To better understand how balancing flexibility capabilities in scheduling timeframes are changing during energy transition, and how these changes might impact the suitability and design of more decentralised operational balancing practices.

The following peer-reviewed journal article published in Energy Policy constitutes this chapter:

Prakash, A., Ashby, R., Bruce, A. & MacGill, I. Quantifying reserve capabilities for designing flexible electricity markets: An Australian case study with increasing penetrations of renewables. Energy Policy 177, 113551 (2023).

7.2 Abstract

Across several power systems with market frameworks, policy-makers are proposing that balancing flexibility requirements emerging during energy transition be addressed through new reserve product markets. However, these may introduce additional costs, constraints and complexity and even encroach upon the functions of existing operational practices. Thus, policy-makers need to assess and compare flexibility design options, and quantifying system flexibility capabilities based on current and expected resource mixes can assist in achieving this. In this article, we offer a practical method to quantify the time-varying spectrum of upwards and downwards flexibility capabilities in systems, and subsequently apply it to historical and projected resource mixes in two regions of the Australian National Electricity Market. Our results suggest that with higher penetrations of renewable energy: 1) downwards flexibility margins can be exhausted around noon if wind and solar are unable or unwilling to provide it, 2) upwards flexibility becomes more scarce during morning and evening peak demand events and 3) a greater portion of upwards flexibility is provided by energy-limited resources. Given these trends, we recommend that policy-makers examine how existing operational practices can be augmented to elicit upwards flexibility provision, and that duration specifications and sustained footroom procurement be considered for reserve products.

7.3 Introduction

The reliable and secure operation of power systems is contingent upon locational and temporal balancing of active power supply and demand. As jurisdictions progressively decarbonise electricity supply through considerable capacity additions of variable renewable energy (VRE) and the retirement of carbon-intensive conventional generation, the nature of short-term risks to system balancing (i.e. those of concern over the range of seconds to days) is changing. The most notable of these short-term risks are (Ela et al., 2011):

• Power system variability, which includes expected changes in the supply-demand balance. Traditionally, variability has been associated with system load movements and fluctuations around pre-determined generator schedules. As energy transition proceeds, system operators (SOs) are becoming increasingly focused on managing variability that arises due to the presence of VRE. This includes the correlated ramping of neighbouring solar PV generation during sunrise and sunset, and that of wind generation following the arrival of a cold front (Australian Energy Market Operator, 2020d; Lew et al., 2013).
• Power system uncertainty, which encompasses unexpected changes in the supply-demand balance. Beyond demand and VRE generation forecast errors, uncertainty also includes singular or widespread outage events that could be the result of a sudden loss of primary energy availability, equipment malfunctions, or common mode failures either triggered by insecure system operation (e.g. significant frequency and/or voltage deviations) or exogenous events (e.g. extreme weather events) (Electricity Sector Climate Information Project, 2021; Matevosyan et al., 2021; Redefining Resource Adequacy Task Force, 2021).

Provided that it is sufficient, leveraging the active power balancing flexibility of a power system (defined by Heggarty et al. (2020) as a system’s “ability to cope with variability and uncertainty”) should enable these short-term risks to be managed. At a particular point in time, the total balancing flexibility capability of a power system is the sum of potential flexibility contributions from resources such as generators, flexible demand and energy storage. However, the flexibility that can actually be deployed at any given time and location is potentially limited by:

1. Physical, economic, social and environmental constraints on the operation of resources (Denholm et al., 2018; Gonzalez-Salazar et al., 2018);
2. Network topology, particularly if deploying a flexibility solution results in the violation of network constraints (Lannoye et al., 2015; Liu et al., 2021); and
3. Operational practices. These include protocols and tools used by the SO (which is ultimately responsible for maintaining supply-demand balance) and electricity market design in power systems with a market overlay (Ela et al., 2016).

Though it is well established that operational practices are crucial to “enabling” balancing flexibility provision (Hirth and Ziegenhagen, 2015; Hsieh and Anderson, 2017; Papaefthymiou et al., 2018), limited attention has been given to assessing the trade-offs between practice changes (Mays, 2021). A typical design choice in power systems with electricity markets is determining whether a balancing function should be performed by the SO, or partially delegated to market participants via market-based mechanisms. Proponents of market-based mechanisms argue that if they are well-designed, their benefit is twofold: appropriate incentives can unlock the efficient utilisation of latent flexibility from existing resources whilst encouraging investment in additional flexibility as a market-signalled need emerges. However, to some extent, desires to maximise market benefits and minimise market distortions need to be weighed against providing the SO with sufficient lead-time and levers to maintain system balance during both normal and extraordinary circumstances (Prakash et al., 2022a; Roques, 2008).

Establishing markets for balancing reserves offers a compromise between SO control and market efficiency (Kristov et al., 2016; Ryan et al., 2014). These enable the SO to set a requirement for, competitively procure and then schedule system headroom (spare generation capacity and potential load curtailment) or system footroom (potential generation curtailment and load increase) with particular power, energy, ramping and quality-of-response (e.g. response time) capabilities (Degefa et al., 2021; Ulbig and Andersson, 2015). Whilst tailored reserve services can be procured through tendering processes, zonal or system-wide markets for reserve products have become increasingly commonplace given that temporal balancing is of greater concern in meshed networks. Additionally, “commodification” of capabilities through products reduces complexity and enables the implementation of auctions, which can improve transparency and competition and be co-optimised with energy or other reserve product markets (Lal et al., 2021; Mancarella and Billimoria, 2021).

The changing nature of short-term risks to system balancing and the accompanying need for greater system flexibility is leading policy-makers to reassess the suitability of the reserve products available to their SOs (Energy Security Board, 2021a; EU-SysFlex, 2019; Federal Energy Regulatory Commission, 2021). Reform of reserve arrangements can simply modify procurement practices or lead to a more significant restructuring of available products, which includes introducing new markets (Ryan et al., 2014). Particularly in their initial stages, reform processes tend to justify changes on the basis of how they might address potential threats to system balancing. This approach is appropriate and sufficient where reserve service provision entails specialised quality-of-response capabilities that cannot be provided effectively or efficiently through other means (e.g. high bandwidth control configurations required for fast frequency response provision). However, some reserve products may “compete” with other design options. For example, the purpose and timeframe of tertiary frequency control and ramping products overlap with those of dispatch processes. Where reserve arrangement reform encroaches on the functions of other processes and practices, quantifying system flexibility capabilities based on current and expected resources mixes can assist policy-makers in assessing flexibility design options.

Reserve products also impose tangible and intangible costs. Regardless of cost allocation mechanisms, procuring reserves typically raises system operation costs and thus prices paid by energy users (Hummon et al., 2013). Furthermore, even if they offer a solution to a system sub-problem, reserve products do not guarantee reliable operation of the overall system and may even hinder the implementation of other measures that can realise system flexibility (MacGill and Esplin, 2020; Papaefthymiou et al., 2018; Pollitt and Anaya, 2019). For example, valuing balancing flexibility on the scale of minutes to hours through reserve products could mean sacrificing the benefits of better reflecting the value of flexibility in energy prices:

1. For participants, energy market risk management is more straightforward than managing risk in reserve product markets. Short-term energy markets typically have greater depth and a broader range of associated technical or financial forward markets (Pollitt and Anaya, 2019).
2. Reserve product markets often have pre-qualification criteria and minimum offer quantities. As such, the participation of smaller demand-side and distributed energy resources (DER) in reserve product markets is often contingent on the involvement of an intermediary aggregator, which imposes additional transaction costs (Poplavskaya and de Vries, 2019). However, embedding the value of flexibility within the price for energy could simplify flexibility provision through market participation for these resources, particularly if policy-makers pursue dynamic retail pricing or nested distribution-level markets that interface with transmission-level markets (Hogan, 2019; Kristov et al., 2016; Mays, 2021).
3. The flexibility that the SO is able to procure through reserve products is restricted by their product specifications. Solely relying on reserve products for flexibility may constrain operational outcomes. Such flexibility “discretisation” might also be reflected in the resources deployed in the system should reserve product markets influence investment decisions (Lal et al., 2021). Additionally, whilst reserve products can be tailored to a particular system’s capabilities and needs, reserve sharing between SO jurisdictions is easier if technical specifications are standardised (Scherer, 2016).

Given these factors, quantification and comparison are therefore needed to assess the role of reserve products, particularly where (Ela et al., 2021; Y. Rebours et al., 2007):

1. Other operational practice or policy changes have the potential to deliver greater and/or more robust flexibility benefits without the additional costs, uncertainty and complexity of new markets; or
2. Current market design or exogenous resource adequacy policies (e.g. firming revenue guarantees or capacity markets) are driving sufficient investment in flexible resources.

A plethora of metrics that quantify different aspects of system balancing flexibility capabilities have been proposed in the literature (Heggarty et al., 2020; Lannoye et al., 2012a; Mohandes et al., 2019). Rather than solely quantifying flexibility capabilities, operational metrics typically compare short-term flexibility capabilities against a flexibility requirement that is set by one of the following or a combination thereof: rules-of-thumb, net load variability, net load forecast uncertainty and/or probabilistic VRE forecasts. While an SO can use these metrics to identify potential flexibility shortages (Zhao et al., 2016), dimension reserve products (Costilla-Enriquez et al., 2023; Dvorkin et al., 2014) or schedule resources (Nosair and Bouffard, 2015), they may be less useful to system designers assessing changes to practices that leverage decentralised decision-making (e.g. energy and reserve product markets). Broader planning-oriented flexibility capability metrics may be more suitable for such purposes. These include traditional resource adequacy metrics (Stenclik et al., 2021), “inflexibility costs” (e.g. additional system costs due to flexibility constraints as explored in Vithayasrichareon et al. (2017)) or “flexibility adequacy” metrics, such as the insufficient ramping resource expectation proposed in Lannoye et al. (2012b). In particular, Lannoye et al. (2012b) uses time-sequential power system operations data to explicitly calculate the balancing flexibility available after resources are dispatched, though valuable chronological information is lost when the time series generated in the study are converted into probability distributions to calculate the insufficient ramping resource expectation. By retaining a degree of this chronological information, our methodology aims to provide electricity industry stakeholders with a better understanding of the time-varying “spectrum” of system balancing flexibility capabilities, and thus assist them in assessing, comparing and designing potential operational practice changes to improve flexibility in power systems with a growing number of variable and energy-limited resources.

In this article, we offer a practical method for quantifying available reserves and footroom (the balancing flexibility that is available after resources are dispatched to meet system demand), and an example of how such quantification can inform flexible electricity market design. We provide simple extensions to the methodology developed by Lannoye et al. (2012b) that account for flexibility contributions from VRE and battery energy storage systems (BESS), and market participants’ aversions to incurring cycling costs. We then use this methodology in a case study in which we quantify time-varying available reserves and footroom in real-world systems: two regions of the Australian National Electricity Market (NEM). Through a 2020 baseline and two 2025 scenarios, we test four key sensitivities in these two regions: the acceleration of large conventional generation retirement, the rate of deployment of VRE and storage technologies, contrasting resource mixes and operational constraints, and greater variability in operational demand. While previous studies have tested the impact of some of these sensitivities on the availability of total system headroom or existing reserve products (Frew et al., 2021b; Hummon et al., 2013; Tanoto et al., 2021), our analysis offers a perspective that is focused on quantifying a time-varying spectrum of flexibility capabilities and thus concerned with the design of operational practices in low-carbon power systems. Our analysis results highlight the underappreciated need to consider mechanisms for procuring footroom, and we proceed to discuss the implications of implementing new balancing products on operational outcomes. Though the NEM is unique in aspects of its operational practices and the balancing risks it faces, the methodology and findings from this study will become increasingly relevant in other jurisdictions given the accelerating deployment of VRE and storage and the progressive retirement of carbon-intensive conventional generation (International Energy Agency, 2021, 2019).

Section 5.4 provides an overview of how balancing flexibility is enabled and procured through the NEM’s operational practices and market design. In Section 5.5, we describe a methodology to quantify available reserves and footroom across deployment horizons for various resource types. Then, in Section 5.6, we quantify the available reserves and footroom in two regions of the NEM for existing resource mixes in 2020 and potential resources mixes in 2025, with two scenarios for the latter. We then use the findings from this case study to explore the role of reserve products in securing balancing flexibility. We conclude by highlighting pertinent findings and recommendations to policy-makers in Section 5.7.

7.4 Flexibility in the National Electricity Market

The Australian National Electricity Market (NEM) is a short-term wholesale electricity market overlaid on a ~5000 kilometre long “stringy” network that services the majority of eastern and southern Australia (Australian Energy Market Commission, n.d.b). In 2021, it saw a peak demand of ~32 GW and total electricity consumption of ~204 TWh (Australian Energy Regulator, 2022a). With no explicit capacity mechanisms or compulsory forward markets, the NEM solely consists of a zonal real-time platform, with market regions corresponding to the states of Queensland, New South Wales (NSW), Victoria, Tasmania and South Australia (SA). Interconnection between market regions is relatively weak and, due to the large distances involved, the NEM is not connected to other bulk power systems (Australian Energy Market Operator, 2019b).

In the subsections that follow, we describe the operation of the NEM with a focus on features and mechanisms that enable or explicitly procure balancing flexibility. In particular, we discuss current reserve arrangements in the NEM in Section 5.4.3 and the proposal to introduce an operating reserve product in Section 5.4.3.1. The policy debate surrounding the usefulness and design of this potential reserve product provides the primary motivation for our case study in Section 5.6.

7.4.1 Market design

7.4.1.1 Real-time markets

The NEM is a central dispatch market that is operated by the Australian Energy Market Operator (AEMO). On the day ahead of delivery, market participants are required to submit non-binding offers for each resource consisting of price-quantity pairs for energy and, optionally, Frequency Control Ancillary Services (FCAS) (described in Section Section 5.4.3) (Australian Energy Market Operator, 2021k). Energy offers can be priced as high as the market price cap (15,000 AUD/MW/hour during the Australian financial year of 2020-2021) or as low as the market floor (-1000 AUD/MW/hour). Negative pricing enables generators to express a preference to either remain online due to significant start-up/shut-down costs or to be dispatched as a price-taker when it is commercially favourable to do so (e.g. to receive remuneration from an offtake agreement). In theory, it also provides investment signals for flexible resources alongside a relatively high market price cap (Orvis and Aggarwal, 2018; Riesz et al., 2016).

On the day of delivery, co-optimised markets for energy and FCAS are cleared every 5 minutes through a security-constrained economic dispatch process, which produces zonal marginal prices for energy and FCAS. There is no formal gate closure in the NEM; participants are able to alter volumes (but not prices) in their offer up to tens of seconds before the delivery interval (Australian Energy Market Commission, 2015; McArdle, 2021a). In 2021, the market settlement period was changed from 30 minutes (the average of prices of the preceding six 5-minute intervals) to 5 minutes to better align settlement with dispatch and pricing (Australian Energy Market Operator, 2022b). Since resources are expected to linearly ramp between one dispatch target and the next, the dispatch process implicitly “procures” some flexibility to manage variability (Australian Energy Market Operator, 2021d; Ryan et al., 2014). As such, the NEM’s dispatch is relatively fast and granular when compared to short-term electricity markets worldwide (Katz et al., 2019; Silva-Rodriguez et al., 2022).

The NEM’s real-time market is also able to elicit balancing flexibility provision from a variety of resources:

• Unlike some North American markets that permit large proportions of the generation fleet to self-schedule (Ela et al., 2016; Orvis and Aggarwal, 2018), generation with a capacity above 30 MW is required to participate in the real-time market and receive dispatch instructions (Australian Energy Market Commission, 2017). This exposes larger utility-scale resources, which make up the bulk of the NEM’s generation capacity, to price signals that somewhat reflect system balancing requirements.
• VRE forecasts used in dispatch can be generated by AEMO or provided by market participants; due to very late gate closure, both are able to incorporate telemetered operational data from the minutes preceding delivery (Australian Energy Market Operator, 2018f, 2016).
• In 2021, a wholesale demand response mechanism was implemented to enable larger loads (aggregated or otherwise) and virtual power plants (VPPs) to directly participate in the energy market¹⁸ (Australian Energy Market Operator, 2020o).

7.4.1.2 Forward markets

In the NEM, forward energy markets are voluntary and primarily consist of the trading of electricity derivatives between market participants. Though market participants can contract over-the-counter, the majority of forward market activity occurs on two market exchanges for standard products for periods up to 3 years out (ASX Energy, 2021; Australian Energy Regulator, 2021). These standard products include quarterly or annual futures, which fix a price for an agreed quantity of energy, and caps, which are essentially call options that enable contract purchasers (typically electricity retailers) to pay no more than the strike price of 300 AUD/MWh for energy at the cost of a premium paid to the seller. Contract markets in SA are considered to be relatively illiquid compared to those in NSW, Queensland and Victoria (Australian Energy Regulator, 2022a). Beyond enabling market participants to hedge real-time market price risk, products traded on the forward markets may ‘discipline’ market participants into offering balancing flexibility to the system. For example, a generating market participant that sells futures and caps is likely to retain some reliable generation capacity in reserve to avoid large payouts in the event of high real-time prices or the failure of their other plants (Riesz et al., 2016).

7.4.1.3 Limitations

Despite the arguably world-leading flexible design of its real-time markets, there are some notable limitations in the NEM and its associated forward markets:

• To date, the balancing flexibility offered by DER has primarily been leveraged through unremunerated, last-resort curtailment of distributed solar PV in SA by AEMO (Australian Energy Market Operator, 2021l) or through aggregated solar-battery VPPs. At the end of 2021, VPPs had a registered capacity of approximately 30 MW (Kuiper, 2022), a small percentage of the ~15 GW of distributed solar PV capacity installed in the NEM as of June 2022 (Australian PV Institute, n.d.).
• Aside from the procurement of footroom that is only deployed following frequency excursions (Section 5.4.3), there are currently no mechanisms in the NEM that remunerate resources for providing sustained downwards flexibility to the system.
• Standard derivative products have remained much the same for decades despite changes in the NEM’s resource mix and market dynamics. In particular, the 300 AUD/MWh strike price of cap contracts does not necessarily reflect a resource’s operating costs (e.g. the price of natural gas or the charging/pumping price for BESS/pumped hydro energy storage). While a demonstration project trialled a market platform for derivatives designed to be sold by flexible resources (e.g. a “Super Peak” contract that enables buyers to hedge morning and evening demand peaks), these are nascent products with small traded volumes to date (Renewable Energy Hub, 2021).
• AEMO has little visibility and no direct oversight over the voluntary forward markets, which are currently operated by the financial services sector. Moreover, even if AEMO did, it would likely be difficult for them to determine how portfolio-based contracting might influence the operation of particular resources (Australian Energy Market Commission, 2020d).

7.4.2 Ahead processes and operator intervention

Through several ahead processes, AEMO regularly publishes forecasted system and market information to assess power system reliability and assist market participant decision-making. The processes most relevant to operational decision-making include the near-term Projected Assessment of System Adequacy (PASA) and pre-dispatch simulations:

• Using forecasts for demand and VRE, a simplified set of forecasted network constraints and participant-submitted resource availabilities and energy constraints, the Pre-Dispatch PASA and Short Term PASA (run every half-hour and hour, respectively) both assess the maximum generation reserves available in each region for the next 7 trading days. PASA outputs include half-hourly available generation and system load forecasts (Australian Energy Market Commission, 2022; Australian Energy Market Operator, 2020p, 2012a).
• Once day-ahead offers have been submitted by market participants, AEMO uses these offers in pre-dispatch processes alongside forecasts for constraints, demand and VRE. Pre-dispatch simulations then produce forecasts for dispatch conditions and regional prices for energy and FCAS. These are run every half hour at half-hourly resolution until the end of the next trading day (pre-dispatch) and at 5 minute resolution for the next hour (5 minute pre-dispatch) (Australian Energy Market Operator, n.d., 2021k). The potential impacts of demand forecast error on regional energy prices and interconnector flows are explored through a sensitivity analysis (Australian Energy Market Operator, 2021m).

Regional balancing stress is indicated by the level of in-market reserves, which is the total offered generation capacity in excess of forecast regional demand¹⁹. Should the Short Term PASA or pre-dispatch processes forecast in-market reserves below specific trigger levels, AEMO must issue market notices that declare forecast Lack of Reserve (LOR) conditions (Australian Energy Market Operator, 2021n). Trigger levels are set by the maximum of either deterministic generation contingencies (i.e. below N-2 for LOR1, below N-1 for LOR2 and no in-market reserves for LOR3), or a particular confidence level of a probability distribution of total forecasting errors generated by a Bayesian Belief Network, which is trained on historical forecast errors and power system conditions (Australian Energy Market Operator, 2018g).

The intention of these ahead process and LOR notices is to provide market participants with information that might elicit a response, such as shifting planned maintenance or rescheduling flexible resources in response to forecasted tight supply-demand balance conditions. However, if more severe LOR2 or LOR3 notices have been issued and AEMO deems that the market response is insufficient by a certain time, AEMO can intervene in the market by issuing directions (manual dispatch), activating emergency reserves procured through the Reliability and Emergency Reserve Trader (RERT) and/or instructing transmission network operators to shed load (Australian Energy Market Operator, 2021n, 2018g).

7.4.3 Reserve products

Formal reserves arrangements in the NEM consist of eight FCAS and the Reliability and Emergency Reserve Trader (RERT). In each dispatch interval, FCAS are procured by AEMO from markets for raise (headroom) and lower (footroom) regulation FCAS, which are used to provide frequency control during normal operation, and three raise and lower contingency FCAS, which deliver their full response within 6 seconds, 60 seconds or 5 minutes following a major imbalance event. The volumes of FCAS procured for each dispatch interval are dynamically determined, with regulation FCAS procurement volumes dictated by power system time error and contingency FCAS procurement volumes typically corresponding to an N-1 contingency. In the absence of regional constraints, FCAS are procured for and from all regions of the NEM. While FCAS provides balancing flexibility through frequency-responsive headroom and footroom, they predominantly respond to intra-dispatch variability and uncertainty with the expectation that deployed resources will be relieved by 5-minute dispatch (Prakash et al., 2022a; Riesz et al., 2015). 5 minute contingency FCAS is an exception, given that its response may be called upon for up to 10 minutes. 5 minute contingency FCAS is currently provided by a diverse range of resources (see Figure 15).

Through the RERT, AEMO can obtain last-resort reserves given between 1 week to 1 year of notice of forecasted in-market reserves shortfalls. While procurement practices vary depending on the notice time, RERT procurement consists of AEMO contracting with out-of-market resources. Following forecast or actual LOR2 or LOR3 conditions and an insufficient market response, AEMO is able to activate RERT reserves (Australian Energy Market Commission Reliability Panel, 2020; Australian Energy Market Operator, 2021o). The RERT provides AEMO with a last-resort mechanism to procure balancing flexibility prior to any potential load shedding. However, resources that provide reserves through the RERT are unable to participate in the real-time market for the duration of their contract. After RERT reserves are activated, market participants are remunerated based on counterfactual pricing (i.e. dispatch without RERT), thus maintaining scarcity pricing and potential signals for investment.

7.4.3.1 Operating reserves product

An inter-dispatch operating reserve product has been proposed in the NEM. It would enable AEMO to procure headroom, which would need to be available to the real-time market within the product horizon, in each dispatch interval. Horizons of 5 minutes and 30 minutes were proposed (Australian Energy Market Operator, 2021p; Energy Security Board, 2021a). Market bodies and participants have raised several potential benefits of an operating reserve product:

1. It could address both inter-dispatch variability and uncertainty. Market bodies consider that the need to address the latter may be more material due to the growing impact of forecast uncertainty on system balancing and the potential for high impact, low probability power system events leading to extraordinary system imbalances (Australian Energy Market Commission, 2021c; Eggleston et al., 2021).
2. AEMO supports a 30+ minute horizon, as a longer timeframe product is likely to have a larger pool of providers and provide participants/AEMO with more lead time prior to any potential market intervention (Australian Energy Market Operator, 2021p).
3. Through reserve constraints and potential scarcity pricing through an operating reserve demand curve (Hogan, 2013), the product could act as an energy ‘price-adder’. This would enable real-time market prices for energy to better reflect consumers’ preference for reliability (Cramton, 2017). Although the NEM’s market price cap is high by international standards, it is generally well below the estimated value of short-term reliability for both residential and non-residential customers in the NEM (Australian Energy Regulator, 2019a). A ‘price-adder’ could also provide sharper investment signals for flexible resources.

The assessment of reserve capabilities to justify this new product has been limited. AEMO has previously analysed ramping capabilities over timeframes greater than 30 minutes (Australian Energy Market Operator, 2020d), the total reserve capacity available within various timeframes across NEM regions and years (Australian Energy Market Operator, 2021p) and regularly forecasts in-market reserves (Section 5.4.2). However, these studies do not consider flexibility capability available after resources are dispatched, or do not explore the time-varying spectrum of this capability. Using the methodology outlined in Section 5.5, we incorporate these elements when quantifying balancing flexibility capabilities in NSW and SA to inform an assessment of the operational benefits of additional balancing products (Section 5.6).

7.5 Modelling available reserves and footroom

To quantify balancing flexibility capabilities, we consider headroom and footroom that can be converted to stable active power output within a particular time horizon. We will refer to these as available reserves and available footroom²⁰, respectively. Though these metrics do not explicitly consider whether resources are frequency-responsive, how long a potential response can be sustained for and whether network constraints restrain flexibility provision, calculating these quantities is broadly useful for understanding the balancing flexibility that could be deployed in a meshed system within operational timeframes (minutes to hours).

7.5.1 Quantifying available reserves and footroom

At a given point in time and for a particular horizon, the available reserves and footroom that a resource can offer are dependent on its operational constraints, its synchronisation status and its active power output. The latter two can be obtained from historical data, or as the outputs of production-cost or market modelling.

Below, we outline a methodology for calculating system-wide available reserves and footroom (Section 5.5.1.5). We adapt the methodology proposed by Lannoye et al. (2012b) to calculate available reserves and footroom from conventional resources (coal-fired, hydro and gas-fired generation - Section 5.5.1.2), and propose simple extensions for calculating available reserves and footroom provided by VRE (Section 5.5.1.3) and BESS (Section 5.5.1.4). The nomenclature used in these sections is described in Section 5.5.1.1.

7.5.1.1 Nomenclature

7.5.1.1.1 Indices and sets

7.5.1.1.2 Time-varying resource parameters

7.5.1.1.3 Static resource parameters

7.5.1.1.4 Computed quantities

7.5.1.2 Conventional resources

The quantities of reserves and footroom that can be made available by conventional resources are dependent on whether the resource is online (non-zero active power output) or offline.

A conventional resource unit is considered to be online if g_{r_c, t} > 0. The reserves that an online conventional resource unit can make available within the horizon h (AR_{r_c,h,t}^{ON}) is given by:

AR_{r_c,h,t}^{ON} = \begin{cases} \mathrm{StartUp}_{r_c} \times h & 0 < g_{r_c,t} < \mathrm{MSL}_{r_c}, h \leq \mathrm{SUT}_{r_c,t} \\ \min(\\ \hspace{0.2cm}(\mathrm{MSL}_{r_c} - g_{r_c,t}) + \mathrm{RampUp}_{r_c} \times (h - \mathrm{SUT}_{r_c,t}),\\ \hspace{0.2cm}\overline{g}_{r_c,t} - g_{r_c,t}\\) & 0 < g_{r_c,t} < \mathrm{MSL}_{r_c}, h > \mathrm{SUT}_{r_c,t} \\ \min(\mathrm{RampUp}_{r_c} \times h, \overline{g}_{r_c,t} - g_{r_c,t}) & g_{r_c,t} \geq \mathrm{MSL}_{r_c} \end{cases}\qquad{(4)}

The three conditions in Equation 4 reflect the following:

1. The unit is in its start-up sequence (i.e. 0<g_{r_c,t} < \mathrm{MSL}_{r_c}) and the reserve horizon (h) is shorter than or equal to the unit’s start-up time (\mathrm{SUT}_{r_c,t}). In this case, the start-up ramp rate (\mathrm{StartUp}_{r_c}) dictates the quantity of reserves that the unit can provide.
2. The unit is in its start-up sequence and the reserve horizon (h) is longer than the unit’s start-up time (\mathrm{SUT}_{r_c,t}). In this case, the quantity of reserves that the unit can provide is the minimum of the total unit ramping potential within the reserve horizon (at rate \mathrm{StartUp}_{r_c} up to the unit’s minimum stable level, and \mathrm{RampUp}_{r_c} beyond it) and the unit’s headroom.
3. The unit is operating above its minimum stable level. The quantity of reserves that the unit can provide is the minimum of the total unit ramping potential within the reserve horizon (at rate \mathrm{RampUp}_{r_c}) and the unit’s headroom.

The reserves that an offline conventional resource unit can make available within the horizon h is given by Equation 5, which has two conditions that resemble the first two conditions of Equation 4:

AR_{r_c,h,t}^{OFF} = \begin{cases} \mathrm{StartUp}_{r_c} \times h & g_{r_c,t}=0, h \leq \mathrm{SUT}_{r_c,t} \\ \min(\\ \hspace{0.2cm} \mathrm{MSL}_{r_c} + \mathrm{RampUp}_{r_c} \times (h-\mathrm{SUT}_{r_c,t}),\\ \hspace{0.2cm} \overline{g}_{r_c,t} - g_{r_c,t}\\) & g_{r_c,t}=0, h>\mathrm{SUT}_{r_c,t} \end{cases}\qquad{(5)}

To ensure that flexibility quantification only considers stable changes in active power output, footroom from conventional resource units is defined to be the maximum downwards flexibility they can provide without shutting down (i.e. down to their MSL). As such, footroom can only be provided by online units operating above their MSL (first condition in Equation 6):

AF_{r_c,h,t}^{ON} = \begin{cases} \min(\mathrm{RampDown}_{r_c} \times h, g_{r_c,t} - \mathrm{MSL}_{r_c}) & g_{r_c,t} > \mathrm{MSL}_{r_c} \\ 0 & 0 < g_{r_c,t} \leq \mathrm{MSL}_{r_c} \end{cases}\qquad{(6)}

7.5.1.3 Variable renewable energy

Within the availability of their primary energy source and the timeframes of concern in this study, VRE are considered to be highly flexible (Holttinen et al., 2021; Nelson et al., 2018). Therefore, the provision of available reserves (AR_{r_v, h, t}) and footroom (AF_{r_v, h, t}) by VRE is not limited by ramp rates but rather by headroom and footroom:

AR_{r_v,h,t}= g^f_{r_v,t}-g_{r_v,t}\qquad{(7)}

AF_{r_v,h,t}=g_{r_v,t}\qquad{(8)}

In this study, g_{r_v,t} < g^f_{r_v,t} can occur as the result of VRE curtailment due to oversupply.

7.5.1.4 Battery energy storage systems

BESS are also highly flexible and, unlike other resource types, can provide additional flexibility by switching from charging (g_{r_b, t} < 0) to discharging (g_{r_b,t} > 0), or vice-versa. This additional flexibility can be accounted for by including the maximum power capacity of the BESS (\overline{g}_{r_b,t}, which restricts BESS charging and discharging such that |g_{r_b,t}| \leq \overline{g}_{r_b,t}) in the equations for available reserves (Equation 9) and available footroom (Equation 10):

AR_{r_b,h,t}=\overline{g}_{r_b,t}-g_{r_b,t}\qquad{(9)}

AF_{r_b,h,t}=\overline{g}_{r_b,t}+g_{r_b,t}\qquad{(10)}

7.5.1.5 System-wide

At time t, the total reserves and footroom that can be made available to the system within the horizon h are given by Equation 11 and Equation 12, respectively:

AR_{h,t}=\sum_{r_c \in \mathcal{R}_c}{(AR_{r_c,h,t}^{OFF} + AR_{r_c,h,t}^{ON})} + \sum_{r_v \in \mathcal{R}_v}{AR_{r_v,h,t}} + \sum_{r_b \in \mathcal{R}_b}{AR_{r_b,h,t}} \qquad{(11)}

AF_{h,t}=\sum_{r_c \in \mathcal{R}_c}{AF_{r_c,h,t}^{ON}} + \sum_{r_v \in \mathcal{R}_v}{AF_{r_v,h,t}} + \sum_{r_b \in \mathcal{R}_b}{AF_{r_b,h,t}} \qquad{(12)}

These equations are used to calculate system available reserves and footroom for all reserve horizons of interest (h \in \mathcal{H}) across all of the dispatch intervals in a given scenario year (t \in \mathcal{T}).

7.6 Case study: two regions in the National Electricity Market

7.6.1 Scenarios

In this study, available reserves and footroom were quantified for NSW and SA in calendar year 2020 and for two resource mix scenarios in 2025 (see Table 1). The 2025 scenarios roughly correspond to the Central and Step Change scenarios in AEMO’s 2020 Integrated System Plan (ISP)²¹ (Australian Energy Market Operator, 2020c), a least-regrets transmission planning study that incorporates scenario-based capacity expansion modelling (Australian Energy Market Operator, 2020h).

Modelling SA and NSW across these three scenarios enables four sensitivities to be explored:

1. Conventional generation retirement. For NSW, one coal-fired power station is retired in 2025 Central and two in 2025 Step Change. In SA, four gas-powered steam turbine (Gas-Steam) units and two combined-cycle gas turbine (CCGT) units are retired between 2020 and both 2025 scenarios.
2. Increasing deployment of VRE and BESS. Additional VRE and BESS capacity is deployed in both states between 2020 and 2025 Central in AEMO’s 2020 ISP. In the 2025 scenarios for both states, a greater quantity of VRE (predominantly solar PV) and BESS is installed in the Step Change scenario than in the Central scenario. The addition of 2 GW hydro generation in NSW by 2025 reflects the expansion of the region’s largest hydro scheme (Snowy 2.0). The capacity mix of each state in 2020 and the changes in the mix for each 2025 scenario are shown in Figure 16.
3. Contrast in resource mix and thus operational constraints. In NSW in 2020, coal-fired generation is a large proportion of the generation fleet and is complemented by hydro generation, gas-fired generation (CCGTs and OCGTs) and VRE. In SA in 2020, VRE (especially wind) is a significant portion of the region’s generation fleet. SA’s synchronous generation consists of gas-fired generation across the flexibility spectrum, some of which must remain online to ensure there is sufficient system strength in SA for secure operation.
4. Greater variability in operational demand due to more distributed solar PV. Operational demand is defined as the system demand that AEMO dispatches resources to meet (i.e. excluding demand met by DER). As the capacity of distributed solar PV in each region increases (i.e. from 2020 to 2025 Central to 2025 Step Change), operational demand in the middle of the day is eroded whilst ramping requirements in the morning (downwards) but especially the evening (upwards) increase. In other words, higher penetrations of distributed solar PV leads to a “deeper” duck curve (Australian Energy Market Operator, 2020l).

7.6.2 Methodology

For each region and scenario, the available reserves and footroom in the system were calculated from the results of a year-long time-sequential market simulation implemented in the commercial electricity market modelling tool PLEXOS (Energy Exemplar, 2021). The PLEXOS market simulation consisted of a PASA phase to model maintenance and forced outages for conventional generation across the year, a Medium Term Schedule phase in NSW to schedule hydro generation according to monthly energy constraints, and a Short Term Schedule phase that carries out unit commitment and economic dispatch (UC-ED) at 5-minute resolution in daily steps²².

Each existing coal-fired (NSW) and Gas-Steam (SA) unit was explicitly modelled to accurately capture the consequences of partial and full outages of large capacity units. For other resource types, the operational constraints and attributes of individual units were averaged and applied across all units of a resource type. This enabled clustered UC-ED and thus reduced the computational burden of the Short Term Schedule phase (Palmintier and Webster, 2014). For baseload conventional generation and gas turbines, ramp rates in each direction were separated into a market ramp rate, which was used in the PLEXOS market simulation, and an upper ramp rate, which was used to calculate available reserves/footroom (Section 5.5.1.2). A lower magnitude ramp rate in the market simulation (market) reflects participants’ preferences to reduce cycling wear-and-tear due to demanding ramping during typical operation (especially for ageing assets) (Kumar et al., 2012), whilst using a higher magnitude ramp rate to calculate a resource’s available reserves and footroom (upper) ensures that the total available flexibility of a resource can be utilised if needed in a system emergency.

Both NSW and SA were modelled assuming a copper-plate network with no interconnection to other regions (i.e. single bus with no network constraints). The Short Term Schedule mixed-integer linear program was solved using the CPLEX Optimizer (IBM, 2021) with a relative mixed-integer program gap tolerance of 0.07%. The generation and synchronisation status of each resource was obtained from the solution and used to calculate the available reserves and footroom for each 5-minute interval using the equations outlined in Section 5.5. A process flow diagram of the study methodology is shown in Figure 17.

In Appendix A, we outline our sources for key input data and assumptions (top row of Figure 17) and provide further details regarding how these data were used in the market simulation and/or the calculation of available reserves and footroom.

7.6.3 Limitations

There are two important caveats to this study. The first is that this study models each region in isolation – that is, resources in other NEM regions can neither assist in meeting demand nor provide available reserves or footroom through cross-regional interconnectors. During typical operating conditions, it is likely that any headroom/footroom on interconnectors would mean that a greater quantity of reserves/footroom are available to a region, albeit at different horizons due to modified dispatch patterns. For example, the inclusion of interconnectors in the SA model between SA and VIC and SA and NSW²³ may increase the total available reserves/footroom in SA at the cost of a decrease in the reserves/footroom available within shorter horizons. This could arise from local mid-merit gas generators remaining offline in favour of inflexible but cheaper coal-fired generation in NSW and VIC.

However, modelling available reserves and footroom for isolated regions may provide a closer approximation to reality when balancing flexibility is scarce in a region. Under these circumstances, it is likely that interconnector flows will already be close to their limits. This will reduce or altogether prevent the available reserves/footroom provision from resources in neighbouring regions. Moreover, large interconnector flows may be prevented if there is a credible risk of regional separation (loss of synchronism between market regions due to interconnector circuit faults — a particular risk in the NEM due to limited interconnection between market regions); at present, AEMO co-optimises interconnector flow with regional FCAS procurement (Australian Energy Market Operator, 2010). An additional consideration is that if an operating reserve product is implemented to improve the NEM’s resilience to supply-demand shocks, regional procurement requirements may also limit the available reserves/footroom that can be procured over an interconnector. As such, the modelling of isolated regions may approximate actual operation when reserves/footroom are scarce and thus most valuable to the system.

The second caveat is that this study does not explicitly model FCAS procurement. If headroom or footroom reserved for FCAS is unable to also provide available reserves or footroom²⁴, then modelling FCAS markets would reduce the reserves and footroom that are available within horizons less than or equal to 5 minutes. However, the actual headroom/footroom reduction would depend upon the following factors:

• Whether regional FCAS procurement constraints bind for the modelled region. If they do not, multi-regional or NEM-wide FCAS requirements can be satisfied by procuring FCAS in other market regions.
• The degree to which headroom/footroom is “re-offered” across sequential FCAS markets. For example, a single resource enabled for 10 MW across the three raise contingency FCAS markets would withdraw less system headroom than three resources enabled for 10 MW each for a particular FCAS market.
• Headroom that is offered into the 6 second and 60 second raise contingency FCAS market may not reflect sustained power provision. For example, frequency response from a steam-powered turbine may draw on steam stored in a boiler; a sustained response would require a longer timeframe due to slower boiler dynamics.

7.6.4 Results and discussion

7.6.4.1 Synthetic daily profiles

Synthetic daily profiles (SDPs) were developed to quantify the time-varying spectrum of available reserves and footroom for each scenario. For a given horizon, the SDP value at a particular time is an aggregate value (mean or a specific percentile) calculated from the reserves/footroom available within that horizon at the end of that dispatch interval across all days in the simulated year. In other words, values from across the year for a given time of day are aggregated, and these are then “stitched” together to form a “synthetic day” curve for a particular horizon. Two aggregate values were calculated for each horizon curve:

1. The mean. This provides a picture of the average or “typical” availability of reserves and footroom at different times of the day for a particular scenario year; and
2. The bottom 1% (i.e. 1^st percentile or 1-in-100 day lowest). This measure better reflects the availability of reserves and footroom when they are scarce and thus when they are most needed²⁵.

In addition to an infinite horizon (which corresponds to the maximum availability), curves were calculated for 1, 5, 15, 30 and 60 minute horizons. These horizons encompass the start-up times of hydro and flexible gas generation, and represent the likely timeframes over which the proposed operating reserve product will be required to respond.

7.6.4.2 Available reserve synthetic days

Mean and bottom 1% available reserve SDPs were generated for the NSW scenarios and for the SA scenarios (Figures 18, 19). The mean SDPs across scenarios suggest that, on average, NSW has more than 2 GW and SA more than 600 MW of reserves available within 5+ minutes. These levels of reserves:

1. Correspond to approximately 15% and 20% of peak demand in 2020 in NSW and SA, respectively. These 5+ minute “reserve margins” (i.e. 5+ minute reserves as a percentage of peak demand) are comparable to lower-end reserve margins anticipated for the summer of 2022 in North American jurisdictions (North American Electric Reliability Corporation, 2022).
2. Exceed the highest N-1 contingency in 2020 (i.e. highest LOR2 trigger level declared in the last run of Pre-Dispatch PASA prior to delivery — see Section 5.4.2) by approximately 225% in NSW and 170% in SA (Prakash et al., 2023b).

Furthermore, with additional BESS and flexible gas resources expected to be deployed, the mean 5+ minute reserve margins of both regions are higher for most parts of the day in the 2025 Step Change scenario. Though the market simulation relied on perfect foresight (additional uncertainty may reduce reserve margins), these results suggest that reasonable quantities of reserves are available in each region within a 5+ minute horizon.

Across scenarios, the following trends are apparent in the SDPs:

1. From 2020 to the 2025 Step Change scenario, a midday peak in the mean available reserves SDPs becomes more pronounced. This can be attributed to the increasing displacement of conventional generation by lower-cost utility-scale solar PV in dispatch (an outcome observed by Hummon et al. (2013) and Tanoto et al. (2021)) and the progressive erosion of daytime operational demand due to higher penetrations of distributed solar PV. Particularly in SA, curtailed VRE and BESS also contribute to this reserve “surplus”. BESS in particular are often charging during such periods of plentiful supply and low prices, and thus are able to offer up to double their active power rating as reserve (i.e. by switching from charging to discharging).
2. As is particularly clear in the bottom 1% SDPs for the 2025 scenarios, the availabilities of different reserve horizons tend to converge during periods of lower reserves or “relative scarcity”, which include peak demand events in the morning and evening. The convergence may be driven by the retirement of baseload conventional generation and higher ramping requirements in the 2025 scenarios requiring more flexible, mid-merit resources to be online prior to and during these periods.

From this analysis, we can also gain an insight into the supply-side dynamics of a potential operating reserve product market. The first trend suggests that as energy transition proceeds, a reserve surplus during the daytime could suppress the price of an operating reserve product (a dynamic that is further explored by Frew et al. (2021b)). Moreover, the convergence of availability across horizons during periods of “relative scarcity” suggests that relatively inflexible but cheaper resources are being preferentially ramped through dispatch at these times whilst more flexible but expensive resources are left in reserve. Since the majority of system headroom during these periods appears to be available within 5 to 15 minutes, operating reserves would likely be procured from these more flexible resources regardless of whether the product requires availability within 5 or 30 minutes. As such, concerns regarding limited providers of a 5-minute horizon product may also apply to a 30-minute horizon product during periods of relative scarcity (noting that several resource types in the NEM are already providing upwards flexibility within 5 minutes in the NEM, as shown in Figure 15).

7.6.4.3 Available footroom synthetic days

Two types of SDPs were constructed for available footroom: one for firm footroom and the other for total footroom. The former refers to potential footroom provision from conventional resources and BESS, whereas the latter also includes footroom that can be provided by curtailing VRE. Figures 20, 21 show mean and bottom 1% SDPs across NSW scenarios for firm footroom and total footroom, respectively. From the bottom 1% SDPs in Figure 20, it is clear that firm system footroom can become very low in NSW in 2025 as remaining baseload conventional generators are driven to operate closer to their MSLs. However, such concerns could be alleviated if VRE provide footroom (Figure 21). A similar result was observed for the SA region.

The available footroom in the system is likely sensitive to extent of conventional generation retirements. Further retirements may enable remaining conventional resources to operate at a higher loading, thereby increasing the available footroom in the system. Regardless, given that each region appears to suffer a lack of firm footroom for several hours during the day in the 2025 scenarios explored in this case study, mechanisms for procuring sustained downwards balancing flexibility should be considered alongside those for procuring sustained upwards balancing flexibility. One simple option would be to implement an operating footroom product, which, if VRE are permitted to provide this service, can enable conventional generation to operate closer to their MSL and thus reduce system operating costs and carbon emissions (Nelson et al., 2018).

7.6.4.4 Short-term energy-limited reserves

While the available reserves metric does not consider the duration for which reserve deployment can be sustained, we can infer whether reserves are short-term energy-limited (i.e. with a duration no more than a few hours) based on their resource type. For this analysis, BESS reserve power was calculated based on the BESS’s state of charge at the end of each dispatch interval and the requirement to sustain provision for 15 minutes. This duration is consistent with the BESS power and capacity that is reserved in SA for the possibility of loss of interconnection (Australian Energy Market Operator, 2020q). In addition, the maximum available price-responsive demand available in each state was added to the available reserves in each dispatch interval (assuming an emergency response time of 5 minutes) to gain a better understanding of the maximum potential contribution of demand response. This corresponded to ~60 MW in SA and ~290 MW in NSW, based on AEMO analysis and forecasts in Australian Energy Market Operator (2020b). Both BESS and DR can be considered to be short-term energy-limited reserve providers. Though conventional generation fuel constraints (e.g reservoir schemes and the gas system) were not modelled in this market simulation, the contribution of conventional resources was separated into those of thermal and hydro to assess the importance of the energy constraints on each resource type to available reserves in NSW.

Tables and show the median percentage across dispatch intervals in a scenario year of available reserves provided by a resources type for NSW and SA, respectively. Whilst hydro and thermal resources dominate 5 minute horizon reserve provision in 2020 in NSW and SA, respectively, short-term energy limited resources provide a greater proportion of reserves in this horizon in 2025. In particular, the median contribution of BESS to reserves available within 5 minutes is 16% for NSW and 40% for SA in the 2025 Step Change scenario. As the reserve horizon is extended to 30 minutes, a greater proportion of reserves are provided by conventional resources, which may be better positioned to sustain a response beyond the short-term²⁶. These results indicate that as energy transition progresses, a trade-off between reserve deployment speed and duration develops. This trend reaffirms the value of the sequential and hierarchical approach to reserve product design and deployment that has been adopted in many jurisdictions (Prakash et al., 2022a). Moreover, it should be noted that unlike other mechanisms for procuring balancing flexibility, reserve services and products can specify duration/energy requirements and thus ensure that flexibility provision is sustained.

7.6.5 The role of balancing products

It is unclear whether introducing an operating reserve product will deliver material operational benefits to the NEM in light of the revenue risks, complexity, and implementation and ongoing costs associated with a new market. Instead, existing mechanisms may be able to deliver sufficient upwards flexibility, particularly if they can be augmented:

1. Market participants with forward market obligations are strongly incentivised to offer balancing flexibility to the market. The premium payment offered to the seller, along with a strong financial incentive to perform during periods of system stress, means that derivatives such as cap contracts somewhat resemble pay-for-performance capacity remuneration mechanisms²⁷. Participants would have further incentive if contracting were made mandatory (Mays et al., 2022), or if they increasingly resort to contracting to hedge pricing volatility that could occur as energy transition progresses (de Vries and Sanchez Jimenez, 2022).

2. Market and system information and forecasts (e.g. the NEM’s ahead processes) may be critical to ensuring that market participants schedule resources to provide flexibility to the system. Future work should not only seek to improve their accuracy and their treatment of uncertainties, but also to understand how they shape participant decision-making and thus which enhancements could provide the most value.

However, there remain some operational benefits of additional balancing products. Nested distribution-level markets and/or real-time market scheduling of aggregated resources have the potential to better enable balancing flexibility from DER. However, a key insight from Section 5.6.4.4 is that consideration should be given to the duration of this flexibility. System stress could coincide with periods in which DER owners wish to use these resources for themselves (e.g. a heatwave or if they are exposed to real-time market volatility to some extent) (Roberts et al., 2020). In contrast, reserve products that specify response durations could provide the SO with certainty that flexibility is only procured from resources that are available for a minimum period of time. Any duration requirements would need to be balanced against the quantity and diversity of flexibility providers – primarily to ensure that product markets are competitive, but also because successive deployment of several short-term energy limited resources may be sufficient to meet system balancing needs over the course of a few hours. Furthermore, sustained footroom products might assist SOs in managing a lack of firm footroom (Section 5.6.4.3). Typically, energy prices rise when upwards flexibility is scarce, thereby compensating providers of upward flexibility. In contrast, downwards flexibility providers are not strictly compensated through energy pricing, as oversupply could lead to dispatch curtailing, rather than remunerating flexible resources. Though this might mean flexible resources avoid financial losses, it comes at the cost of footroom available to the system. Accordingly, an “operating footroom” product that remunerates downwards flexibility offers a solution to the tension between dispatch incentives and the need for system footroom.

7.7 Conclusion and policy implications

State-of-the-art resource adequacy assessments are closing the gap between traditional capacity adequacy assessments, which focus on capacity reserve margins during peak demand events, and flexibility adequacy assessments that often model chronological operations (Stenclik et al., 2021). Yet flexibility adequacy assessments alone do not necessarily offer a better understanding of what type of balancing flexibility a system has and might need, and how best to make it available to the system. As resource mixes change dramatically during energy transition, system designers, planners and operators should quantify balancing flexibility capabilities to gain an appreciation of the availability of different resource types to inform operational practice design.

By quantifying balancing flexibility “margins” in two sub-systems of the Australian National Electricity Market (Section 5.6), we identify potential balancing flexibility dynamics and trends in future power systems. Firstly, systems with high penetrations of distributed and utility-scale solar PV will likely have reserve “surpluses” around the middle of the day and periods of relative reserve scarcity during morning and evening peak demand events. In such systems, the periods when reserves are most valuable do not necessarily correspond to the periods during which it is most efficient to curtail renewable energy generation (due to oversupply or to obtain reserves). As such, a key recommendation for policy-makers is to consider whether reserve product markets are needed to elicit sufficient balancing flexibility provision during these short periods of relative scarcity, or whether adjusting energy market settings, forward market obligations and/or market and system information processes can achieve this. Understanding the potential benefits of new reserve product markets is crucial because they can introduce additional costs, constraints and complexity whilst encroaching upon the functions of other operational practices. Secondly, our study highlights the importance of placing a greater emphasis on duration, as resources touted as essential future balancing flexibility providers (e.g. battery energy storage, demand response) may only be able to sustain a response for at most a few hours. Thirdly, we highlight the need to consider footroom and the benefits of enabling renewable energy to provide it. Footroom procurement and response duration specifications are underappreciated by prevailing market designs, and may be better addressed by policy-makers either modifying existing or creating new reserve product specifications.

8 The scheduling role of future pricing information in electricity markets with rising deployments of energy storage: a National Electricity Market case study

8.1 Link to thesis

Market participation decisions and thus resource schedules are informed by knowledge processes, which provide current and forecasted power system and market information. As such, these knowledge processes and, more broadly, market participation rules must be purpose-fit to enable resource scheduling that leads to effective and efficient system balancing. This chapter explores how market information (specifically in this case study, the centralised price forecasts generated by the system and market operator in the Australian National Electricity Market) and market participant operational strategies impact the deployability of balancing flexibility from energy-limited storage resources, which are expected to aid in balancing electricity markets with high penetrations of variable renewable energy through energy arbitrage. The purpose of this chapter is to address the third research objective of this thesis:

To explore how more decentralised operational balancing practices can be configured to maximise the deployability of balancing flexibility in scheduling timeframes.

The content of this chapter is from the following manuscript submitted to Energy Economics:

Prakash, A., Bruce, A. & MacGill, I. The scheduling role of future pricing information in electricity markets with rising deployments of energy storage: an Australian National Electricity Market case study. Submitted to Energy Economics and under review

8.2 Abstract

In wholesale electricity markets, resource schedules result from market participant decisions informed by knowledge processes, which provide current and forecasted power system and market information. Ensuring that these knowledge processes and market participation rules are purpose-fit is becoming increasingly important with growing deployments of energy storage resources expected to aid in balancing high renewables power systems through energy arbitrage.

Our study explores the scheduling coordination role of centralised price forecasts generated by the system and market operator in the fast, flexible and volatile Australian National Electricity Market. Our work offers three contributions: (1) highlighting the increasing frequency and severity of errors in these forecasts, and proposing a hypothesis that market participant (re)bidding is partially responsible for this phenomenon; (2) modelling the extent to which arbitrage revenues might be reduced should these forecasts guide battery energy storage scheduling; and (3) discussing potential changes to participant scheduling strategies and market design that could improve scheduling outcomes. We recommend that Australian policy-makers not only increase the frequency at which centralised knowledge processes are run, but also consider whether stricter market participation restrictions might incentivise participant bidding strategies that are less likely to induce sudden price swings that can hamper effective scheduling.

8.3 Introduction

Effectively and efficiently scheduling electrical power system resources (generators, loads, energy storage and network elements) is crucial to achieving the primary goal of power system operators: least-cost active power balancing subject to technical constraints. Resource scheduling procedures have changed not only with shifts in resource mixes (Ela et al., 2016; Orvis and Aggarwal, 2018) and advances in information technology and algorithm design (Isemonger, 2009; Knueven et al., 2020), but also where jurisdictions have restructured their electricity industries to replace vertically-integrated utilities with competitive wholesale electricity markets (Chow et al., 2005). Such restructuring has produced three shifts in the scheduling process (Chao et al., 2005; Sioshansi, 2006; Wood et al., 2014). Firstly, resource information became more distributed since only market participants know their resources’ true status, operational constraints and cost structures. Thus system operators, which are typically no longer vertically-integrated, may have limited visibility of system resources. Secondly, decision-making has become more decentralised, with market participants choosing how they participate by submitting offers, bids and technical constraints that may or may not be truthful²⁸. Thirdly, and most pertinent to this work, resource scheduling decisions are no longer solely made by the system operator based on system conditions; instead, market participants’ scheduling decisions are primarily motivated by prices for energy and other system services.

Several economists (notably F. A. Hayek) have argued that the price system is a preferable coordinating mechanism for production and planning where “dispersed bits of incomplete and frequently contradictory knowledge” are possessed by diverse actors (Hayek, 1945, p. 519; Littlechild and Kiesling, 2021). While we do not assess the merit of this argument in this article, we adopt its underlying logic: that markets are “mechanisms for collecting, processing and disseminating relevant information” (Von Der Fehr, 2013, p. 93). In the context of electricity markets, this can be understood through two almost inseparable activities that together lead to the collation and transmission of information through prices. Firstly, a market participant submits a set of offers/bids that reflects their knowledge, operational objectives, and technical and commercial positions. Secondly, the information in offers/bids is then integrated into projected or binding prices. These two activities occur iteratively in price formation, a dynamic process in which market participants continuously adjust their participation in response to new information (Bowles et al., 2017). Achieving an equilibrium (optimal or otherwise) is not guaranteed; it is a special outcome that arises only from convergence in price formation (Creative Energy Consulting, 2020).

From this perspective, resource schedules are the outcome of an information aggregation and exchange process mediated by binding and forecasted prices in the electricity market. Assuming that prices reflect the system’s condition and needs²⁹, then appropriate and frequently-run knowledge processes are a prerequisite of the iterative participation that is necessary (but not sufficient) for convergence towards a set of resources schedules that deliver good social welfare outcomes. These knowledge processes can be privately-run (i.e. by market participants or their information consultants) or centralised for the purpose of disseminating consistent public information (e.g. those run by a public meteorological service or the system operator) (Mays et al., 2022; Schweppe et al., 1988; Von Der Fehr, 2013).

Though frictions in knowledge and market participation processes are problematic for optimising the schedules of all power system resources, they are arguably more so for energy storage resources (ESRs) with diurnal storage durations (i.e. < 12 hours). These include most battery energy storage systems (BESS) and some pumped hydro storage schemes (Frazier et al., 2021). Whilst diurnal ESRs are flexible and excel at providing capacity, balancing and other system services (Chernyakhovskiy et al., 2021; Holttinen et al., 2021), opportunity-costs that arise from their energy limitations make decisions regarding how and when to participate harder for diurnal storage than for other resource types (McPherson et al., 2020). Furthermore, to maximise revenues from standalone storage operation, market participants must schedule charging, which replenishes the stored energy necessary to provide more lucrative raise/upwards market products and services, during periods of low (or negative) energy prices³⁰.

In this article, we explore the implications of using “forecasts”³¹ generated by centralised knowledge processes to schedule ESRs with diurnal storage durations in the fast, flexible and volatile Australian National Electricity Market (NEM). Developments in the NEM will be of broad interest given its high penetration of variable renewable energy (VRE) (demonstrated by a record maximum instantaneous renewable energy penetration of ~73% in Q4 2023) (Global Power Energy, 2023), a considerable and growing number of ESRs, and because its real-time market design has long included several features – namely short market intervals, market gate closure close to delivery and a high price cap – that policy-makers elsewhere have considered or implemented in response to the proliferation of VRE in their jurisdictions (International Renewable Energy Agency, 2017; Katz et al., 2019; Papavasiliou, 2020; Silva-Rodriguez et al., 2022).

Our work offers three contributions to the literature on ESR scheduling under imperfect foresight and future electricity market design. Firstly, we analyse historical centralised price forecast data from the NEM to show that the frequency of divergence in these forecasts has increased in recent years, and that supply-driven extreme price swings can occur suddenly. We subsequently examine market participant (re)bidding data and suggest that these phenomena could be partially explained by both the greater number of flexible resources with automated bidding capabilities and a NEM market participant’s ability to continuously rebid up until a few seconds before delivery. Secondly, we use the same price forecast data from a recent year to assess the impact of divergences and price swings on battery energy storage system (BESS) wholesale energy market arbitrage revenue – a metric that arguably reflects the ESR’s contribution to system balancing. In particular, our assessment tests the sensitivity of arbitrage revenues to parameters such as the BESS’ storage duration (ranging from 15 minutes to 4 hours), the length of the forecast lookahead (i.e. how far the scheduler looks into the future) and different objective functions (i.e. how the scheduler interprets arbitrage opportunities). Our results show that using these price forecasts to schedule BESSs can materially reduce arbitrage revenue from what would be achievable with perfect foresight. Thirdly, we discuss market participant scheduling options and potential changes to centralised knowledge processes and market design that could improve resource scheduling outcomes in the NEM and other electricity markets with increasing penetrations of VRE and storage.

Section 6.4 provides an overview of market information, participation and clearing processes in the NEM in addition to context on grid-scale ESR deployment, operation and market participation to date. In Section 6.5, we present a two-part case study of the NEM. We first examine errors in system operator-produced operational price forecasts and propose a hypothesis to explain increasing divergence and the occurrence of price swings in Section 6.5.1. Then, in Section 6.5.2, we use the same centralised price forecasts to schedule a variety of BESSs for arbitrage in one NEM region to assess the impact of price forecast errors (and more broadly, imperfect foresight) on arbitrage revenue. Based on our findings from Section 6.5, we discuss the advantages, disadvantages and feasibility of changes to market participant scheduling, centralised knowledge processes and market design that could maximise the balancing value of resources in Section 6.6. We conclude by highlighting pertinent findings and recommendations to policy-makers in Section 6.7.

8.4 Context

8.4.1 Australian National Electricity Market

The Australian NEM is a short-term wholesale electricity market operated over an electrically-isolated and “stringy” power system that covers the eastern and southern seaboards of the continent. It served approximately 80% of all electricity consumption in the country (~204 TWh) and saw a peak demand of ~32 GW in 2021 (Australian Energy Regulator, 2022a; Department of Climate Change, Energy, the Environment and Water, 2023). Whilst market participants can voluntarily trade electricity derivatives in forward markets (Australian Energy Regulator, 2022a) and tender for longer-term investment support for VRE and storage resources from a growing number of state and federal government schemes (Billimoria and Simshauser, 2023), these are designed and operated around the central pillar of the NEM: its real-time wholesale spot market platform.

In the following sections, we provide a brief overview of the NEM and the centralised knowledge processes that are integral to the NEM’s operation.

8.4.1.1 Real-time markets

The Australian Energy Market Operator (AEMO), which is both the system and market operator, clears a gross pool energy market and 10 voluntary frequency control ancillary services (FCAS) markets³² every 5 minutes through a co-optimised and security-constrained economic dispatch process. For each dispatch interval, this process produces resource dispatch targets (i.e. power output levels and for FCAS providers, headroom/footroom obligations) and zonal energy and FCAS prices for market regions that correspond to the country’s five most densely populated states: Queensland (QLD), New South Wales (NSW), Victoria (VIC), Tasmania (TAS) and South Australia (SA). Market prices can be between the NEM’s market floor (-1,000 AUD/MW/hr) and its market cap (15,100 AUD/MW/hr in the Australian financial year 2021-22), which is one of the highest in the world (Silva-Rodriguez et al., 2022). The energy market used to be settled using the average price of the six dispatch intervals in each half-hourly settlement period; however, in October 2021, the frequency of energy market settlement was changed to match that of pricing through the 5 minute settlement (5MS) rule change (Australian Energy Market Operator, 2022b).

The NEM is a semi-centralised electricity market (Ahlqvist et al., 2022). Specifically, though its mandatory gross pool auctions produce resource dispatch obligations, market participants are given free rein to self manage resource commitment and market participation decisions through a bidding process based not on (verifiable) costs, but prices (Conejo and Sioshansi, 2018; Katona et al., 2023). As a minimum requirement, market participants must submit resource-specific bids (for loads) and offers (for generation) by 12:30 PM Australian Eastern Standard Time on the day ahead of delivery for one or more of the NEM’s real-time markets (Australian Energy Market Operator, 2021q). Bids and offers are sealed (i.e. not publicly available until the day after delivery) and consist of up to 10 price-quantity bands, with market participants submitting a single set of price bands for the entirety of the next trading day and a set of quantities for each dispatch interval in the next trading day³³. Market participants must also submit technical parameters for their resources, including resource maximum availability and ramp rates (Australian Energy Market Operator, 2020r).

Whilst market participants cannot change band prices after the day-ahead deadline, they can modify the quantities they offer in each band via a “rebid”. Market rules dictate that market participants must “as soon as practicable” submit rebids that are not “false or misleading” with a “brief, verifiable and specific reason for the rebid” (Australian Energy Market Commission, 2023a; Australian Energy Regulator, 2019b). However, there are few restrictions on rebidding reasons (e.g. responding to a change in forecast prices is acceptable)³⁴ and since the NEM has no formal gate closure, market participants can rebid until the NEM dispatch engine begins to run for the relevant delivery interval (typically tens of seconds prior to its commencement) (McArdle, 2021a).

8.4.1.2 Market knowledge processes

Given that the NEM’s real-time auctions are blind and because it lacks day-ahead or intraday markets that impose physical and/or financial obligations on market participants, AEMO is responsible for running ahead-of-delivery knowledge processes that assess system reliability and generate information to assist participants in making decisions in the market. These knowledge processes include the Projected Assessment of System Adequacy (PASA) and pre-dispatch processes. With inputs including forecasted network constraints and participant-submitted resource availabilities and energy constraints, resource adequacy information is produced for the next two years by Medium Term PASA’s production cost models, which use demand and VRE traces from several reference weather years, and for the next week by Pre-Dispatch PASA and Short Term PASA, which use demand and VRE forecasts in a simple system reliability model (Australian Energy Market Operator, 2021r, 2012a). The half-hourly demand, VRE, available generation and reserve forecasts published by the shorter-term PASA processes are particularly useful for market participants scheduling resources. Pre-dispatch processes, on the other hand, use the latest set of market participant offers alongside system forecasts in a modified version of the NEM’s dispatch algorithm to generate regional energy and FCAS price forecasts and sensitivities that predict the impact of demand forecast errors on prices and power flows between market regions³⁵ (Australian Energy Market Operator, 2021m). These are published at half-hourly resolution until the end of the next trading day (30MPD) and at five minute resolution for the next hour (5MPD) (Australian Energy Market Operator, 2021k). Both PASA and pre-dispatch processes perform a reliability function in addition to providing market participants with market information; if forecast reserves fall below certain trigger levels and AEMO deems the market response to be insufficient by a certain time, it can take emergency actions to ensure the system remains balanced (Australian Energy Market Operator, 2021n, 2018g).

AEMO refers to the prices generated by pre-dispatch processes as “forecasts”, but they cannot strictly be considered as such since one of their purposes is to provide a signal that elicits market responses from market participants (Hu et al., 2005). For example, a high initial pre-dispatch price might motivate a market participant to commit or reschedule their resources through inframarginal rebids, but doing so could then depress the price in the next pre-dispatch run. This may, in turn, drive the same or other market participants to adjust their participation preferences (and so on). As such, the NEM’s iterative pre-dispatch processes in theory provide a platform for coordinating privately-owned resources in the production of a system schedule (Creative Energy Consulting, 2020). However, in practice, prices may not necessarily converge as real-time approaches, leading to less efficient market participation decisions and less effective scheduling outcomes. As demand and VRE forecasts generally improve closer to real-time, any divergence in pre-dispatch forecasts run close to the delivery interval is likely to be a result of market participant rebidding, sudden outages and/or differences between the constraints in pre-dispatch and actual dispatch (Australian Energy Market Operator, 2021k).

8.4.2 Energy storage resources

Earlier studies predominantly focused on evaluating the potential system or market value of grid-scale ESRs in the presence of market volatility and/or growing VRE penetrations (Bradbury et al., 2014; de Sisternes et al., 2016; McConnell et al., 2015; Sioshansi et al., 2009). Since these studies were undertaken, ESR deployment has accelerated worldwide due to declining BESS costs (especially for lithium-ion chemistries) (International Renewable Energy Agency, 2019; Mauler et al., 2021), the growth in market participation opportunities (which in some cases can be attributed to enabling market reform) (Australian Energy Market Commission, 2021d; Federal Energy Regulatory Commission, 2018b), and policy-makers promoting investment or directly investing in storage to support the reliable operation of their power systems as they decarbonise (International Energy Agency, 2022). The fast pace of ESR deployment in recent years has motivated work to better understand how their value can be realised or limited by who the storage operator is and what scheduling strategy they select (Shan et al., 2021).

8.4.2.1 Factors influencing scheduling

Previous work has explored three factors that have a strong bearing on how ESRs are self-scheduled³⁶ and thus the degree of assistance they provide in balancing the system.

8.4.2.1.1 Which objective(s) the storage operator optimises for

ESRs are technically capable of providing a range of services that grows wider as the point of installation approaches energy consumers (Fitzgerald et al., 2015). Though ESR investment cases typically rely on “value-stacking” several of these services, constraints of a technical, organisational and/or regulatory nature mean that ESR scheduling in operational timeframes often involves trade-offs between the services the resource can provide simultaneously and thus the objectives it can fulfil (Keck and Lenzen, 2021; Ransan-Cooper et al., 2021). Moreover, an ESR’s operational objectives are likely to be dictated by the objectives of the storage operator’s broader portfolio. For example, Wang et al. (2017) demonstrate that an ESR can be scheduled to minimise a load-serving entity’s day-ahead deviation liabilities, Loisel and Simon (2021) show how pumped hydro storage is used to support nuclear resources within the portfolio of France’s largest electricity producer, and Billimoria and Simshauser (2023) explore the impact of storage contracts and their design on the alignment between financial incentives for market participants and system needs in operational timeframes. A particularly important line of work in this area has studied the impact of conflicting market participant and system objectives. Rangel (2008), Sioshansi (2014) and Shan et al. (2021) examine instances in which market participants can use ESRs strategically (i.e. to maximise portfolio profits) to the detriment of the system and social welfare.

8.4.2.1.2 Storage operator preferences and attitudes

Beyond the choice of which services the ESR should provide, the preferences and attitudes that affect ESR operation include the storage operator’s risk appetite and their ability (as well as their willingness) to take or make market prices. Shafiee et al. (2016) show that accounting for an ESR’s price-making potential in its scheduling algorithm may lead to the resource curtailing its discharge during high energy prices to avoid “cannibalising” its own revenues. Similarly, Yurdakul and Billimoria (2023a), using a stochastic risk-based scheduling algorithm to model the decisions of a risk-averse storage operator in the face of pre-dispatch forecast uncertainty, demonstrate that storage operator risk aversion can lead to reduced energy availability and/or curtailed discharge during periods of high prices. Such examples of withholding demonstrate that even during periods of relative scarcity, the full capabilities of an ESR may not always be available to the system.

8.4.2.1.3 What information is considered by the scheduler, and how

All operational strategies heavily rely on information about the current and future states of the system and the market (Abdulla et al., 2018). Resource operators must balance the benefits of processing potentially large and diverse datasets from multiple providers with the advantages to decision-making that arise from only using the most pertinent and tractable information for scheduling (Von Der Fehr, 2013). Whilst storage operators predominantly rely on system and/or market forecasts to schedule their ESRs, there still exists a tension between information quantity and quality when considering how forecasts should be used in scheduling.

On one hand, operators can simultaneously consider multiple possible outcomes (forecast distributions or scenarios) by formulating scheduling as a stochastic or robust optimisation problem. Studies have used these methods to schedule ESRs in markets with a real-time platform (Yurdakul and Billimoria, 2023a) or two (day-ahead and real-time) platforms (Krishnamurthy et al., 2018; Wang et al., 2017), and to size and operate an ESR that can assist in system balancing given a historical distribution of VRE generation forecast errors (Baker et al., 2017). However, beyond the difficulties that arise in characterising scenarios and distributions (e.g. assigning scenario probabilities or, as discussed by Baker et al. (2017) and Yurdakul and Billimoria (2023b), accounting for heavy-tailed forecast error distributions), robust methods can be overly conservative whilst stochastic methods require careful selection of risk metrics and, due to their computational complexity, may even prevent market participants from responding to new information in a timely fashion (Roald et al., 2023; Yang et al., 2022).

Given these issues, storage operators might instead prefer to implement optimal control strategies, such as model predictive control, that reschedule ESRs at regular intervals using the most recent (point) forecast information available. Scheduling decisions made using such approaches are often sensitive to the frequency of control decisions, the length of the control horizon lookahead, the operational characteristics of the ESR and the nature and accuracy of the forecasts used. Though previous studies have investigated the impact of a subset of these factors on ESR scheduling and market revenues, they synthesise forecasts by manipulating historical price time series (Connolly et al., 2011; Dunbar et al., 2014; Sioshansi et al., 2009) or by combining day-ahead and real-time prices (McPherson et al., 2020). As such, the resulting forecasts may not adequately represent those that an market participant might use to schedule an ESR in a short-term electricity market. Two earlier studies from the NEM use realistic forecast data and test the sensitivity of arbitrage revenues to pumped hydro energy storage durations (McConnell et al., 2015; Williams et al., 2019), but only simulate operation on a half-hourly basis (rather than every dispatch interval or 5 minutes). Furthermore, Williams et al. (2019) remove high price events that, as we show in Section 6.5.2, can significantly impact ESR revenues. These factors, combined with the age of these studies, mean that their findings may not necessarily hold for fast real-time electricity markets with high penetrations of VRE and storage.

8.4.3 Participation in electricity markets

Hydro ESRs have long participated in and even shaped the design of several electricity markets across the world (Rangel, 2008). However, it is grid-scale lithium-ion BESSs with diurnal storage durations that have accounted for a large proportion of the growth in ESR deployment in recent years. To date, these BESSs have predominantly participated in ancillary services markets due to their lucrative returns (Rangarajan et al., 2023; Schmidt et al., 2023). In fact, they became the largest provider of FCAS by technology type in Q2 2023 in the NEM (Australian Energy Market Operator, 2023e), and in Texas, they provide the majority of the system operator’s requirements for some reserve products (Mago, 2023). However, despite the greater forecasting risks and cycling demands, arbitrage is likely to become a more important revenue stream for these resources given the potential for growing penetrations of VRE to increase short-term energy market price volatility (Ballester and Furió, 2015; Blazquez et al., 2018; de Vries and Sanchez Jimenez, 2022) and the relative “shallowness” of ancillary services markets, which are already seeing increased competition (Prakash et al., 2023a). A shift towards arbitrage revenue is presently occurring in the NEM; daily price spreads (and thus arbitrage revenue opportunities) have increased on average across all mainland regions over the last decade (Figure 22), and the arbitrage proportion of the estimated gross fleet-wide BESS revenue in Q2 2023 was 57% (up from ~20% in Q2 2021) (Australian Energy Market Operator, 2023e, 2022e).

8.5 Pre-dispatch and its impact on storage scheduling in the National Electricity Market

8.5.1 Pre-dispatch price forecast errors

Aside from what appears to be anomalous outcomes for some months in 2016 (one of which is likely related to the SA system black that year), Figure 23 shows that there were relatively few significant pre-dispatch price forecast errors (i.e. errors with magnitude \geq 300 AUD/MW/hr)³⁷ in the NEM from 2012 to the end of 2017. However, price forecast errors have increased in frequency since 2018 in the day-ahead timeframe, and since 2019 for forecasts published up to 2 hours and 15 minutes ahead of delivery. The rise in price forecast errors consists not only of a greater number of negative price errors (red), which are an expected outcome of the pre-dispatch process (i.e. market participants increase resource availability in response to a high price forecast and subsequently depress the cleared price), but also positive errors, which appear to constitute a large portion of the errors occurring within 15 minutes of delivery. Positive errors this close to delivery are particularly concerning to market participants because they may lead to missed opportunities or, if market participants are scheduling ESR charging or have sold forward contracts that their resources are not positioned to defend, unexpected costs. There does not appear to be a clear link between the occurrence of a major system or market event and a greater number of errors – there were a large number of errors following a system black event in SA in 2016 and the explosion of a QLD coal unit in 2021, but lulls in errors also followed the retirement of a large VIC coal plant in 2017 and a double-separation event in 2018.

Flexible resources such as BESSs, hydro and VRE are able to adapt to changing conditions so long as their primary energy source is available and the market information relevant to their participation is published. To better ascertain whether flexible resources might also be affected by price forecast errors, we examine errors from the last available pre-dispatch price forecast (i.e. from the last P5MIN iteration, which is nominally run 5 minutes before delivery). Figure 24 shows that for NSW in 2021, there were some intervals in which the last available price forecast was different from the actual dispatch price by 10,000–15,000 AUD/MW/hr. Price swings of this magnitude could have a significant impact on inflexible and flexible resource alike, and are possible with only small changes in supply or demand since resource offers and aggregate supply curves in the NEM typically resemble hockey-sticks (Energy Synapse, 2020; Hurlbut et al., 2004). Factors that modify available supply, rather than demand, between the last 5MPD run and actual dispatch are likely to be responsible for these large swings as many of the significant price errors in Figure 24 occurred when demand forecast errors were close to zero. Because sudden outages are only occasional and since AEMO uses telemetered operational data in their renewable energy forecasting systems and regularly monitors pre-dispatch constraint accuracy to check if improvements to their formulation can be made (Australian Energy Market Operator, 2023f, 2016; Prakash et al., 2020), we conjecture that market participant rebidding is a major contributing factor to the large price swings in Figure 24. In the next section, we analyse trends in market participant rebidding and propose a hypothesis that links rebidding activity to the rise in price forecast errors in the NEM.

8.5.1.1 A rise in rebidding: autobidders?

Using data zipfile size as a proxy for the total number of bids, offers and rebids (which we will collectively refer to as rebids hereon for the sake of brevity), Figure 25 shows a gradual increase in the number of rebids beginning in 2018, which roughly coincides with the NEM’s first grid-scale BESS starting to participate in dispatch, followed by a more dramatic increase starting in 2021. Whilst 5MS-related changes likely contributed to the rise in rebidding in 2021, the number of rebids has continued to steadily increase as of the end of 2022. We surmise that this sustained rise in rebidding activity is a result of resources increasingly being commissioned with or adopting automated bidding systems (or autobidders) that can rapidly rebid (McArdle, 2021b).

To better understand which technologies are driving rebidding activity, we calculated the percentage of rebids from each technology type in June of every year from 2013 to 2021³⁸. Figure 26 shows that rebids in June have grown considerably; there were ~1–3 million rebids in June in 2013–2018, but this increased to approximately 4.5 million rebids in 2020 and 46 million rebids in 2021. Though conventional resources (especially hydro) were still responsible for a large share of the increased rebidding activity in June 2021, BESS and wind resources overtook coal and gas-fired power station in the quantity of rebids submitted and BESS and VRE together accounted for approximately 35% of all rebids in June in 2021. This figure increases to just over 40% of all rebids if we include other newer market entrants such as demand response and virtual power plant aggregators. It is worth highlighting that BESS resources in particular are playing an outsized role in rebidding; 66 wind farms and 52 open-cycle gas turbine units accounted for approximately 15% and 9% of all rebids in June 2021, respectively, compared to only 7 BESS plant accounting for ~13%.

Though we cannot make definitive conclusions, we surmise that the significant increase in the number of rebids and the greater role of newer market entrants in rebidding can be partially attributed to the growing use of autobidders in the NEM. Autobidders are typically integrated with BESS market participation control algorithms, and VRE resources are increasingly using them to manage the various complexities of market participation, including more frequent negative pricing (Australian Energy Market Operator, 2023g, 2021s; Ekistica, 2021). Furthermore, we propose that the rise in pre-dispatch price forecast errors, be they some time ahead of delivery or a sudden divergence close to real-time, could at least be partially explained by greater rebidding activity. Previous work that analysed the sensitivity of price outcomes to pre-dispatch inputs from 12 hours ahead demonstrated that changes in bids and offers play a greater role in producing large price forecast errors than changes in demand or renewable energy forecasts (Gorman, 2024). We suggest that market participants could be inducing large price forecast changes between pre-dispatch runs, or between pre-dispatch and dispatch itself (e.g. Figure 24), by regularly and rapidly shifting supply availability through rebids – a participation strategy that is enabled by resource flexibility and facilitated by the use of autobidders. In the next section, we shift focus from analysing pre-dispatch price forecast errors to assessing their impact on the arbitrage revenues of various BESS ESRs.

8.5.2 Energy storage scheduling using pre-dispatch price forecasts

Through an optimisation modelling framework, we test the degree to which information quality affects the annual arbitrage revenues of 100 MW BESSs with different storage durations, scheduling optimisation objectives and scheduling lookaheads. We do so for the NSW market region, which provides an interesting case study given its relatively low price volatility historically (Figure 22) yet recent experience with sudden and significant price swings (refer to Prakash (2023a) and Figure 24).

8.5.2.1 Methodology

8.5.2.1.1 Price data

Two types of price data for the NSW market region from 2021 were used in this study:

1. Actual price data obtained using NEMOSIS to represent perfect price information (Gorman et al., 2018); and
2. Price forecast data generated from processing 30MPD and 5MPD price forecasts obtained using NEMSEER (Prakash et al., 2023b) in three steps. Firstly, because 30MPD only produces forecasts with half-hourly resolution, 30MPD price forecasts were imputed using the next observation carried backwards (i.e. forecasted price at 13:30 is applied to intervals ending at 13:25, 13:20, …, 13:05). This reflects a typical interpretation of a 30MPD forecast. Secondly, because 30MPD is only run every half hour, the latest set of 30MPD forecasts were carried forward (i.e. set of forecasts generated at 13:30 are also used in intervals ending at 13:35, 13:40,…,13:55). Finally, the 30MPD forecasts for dispatch intervals within one hour of delivery were removed in favour of 5MPD price forecasts at 5-minute resolution. Together, these processing steps produced forecasts (or imperfect information) for each dispatch interval with 5-minute resolution up to 15 hours out from delivery³⁹.

8.5.2.1.2 Schedules

Three year-long arbitrage schedules consisting of decisions to charge, discharge or idle were generated for each modelled BESS: one using a perfect foresight model, in which BESS operation is optimised across the entire year using actual price data in a single step, and two (one with actual and one with forecast price data) using a receding horizon optimal control (RHOC) simulation (Figure 27). The RHOC simulations consisted of steps every 5 minutes in which BESS operation is optimised for the duration of the lookahead horizon with actual or forecast price data, but only the action for the next dispatch interval is considered to be binding and is thus retained as a scheduling decision that affects the BESS’s state-of-charge. Successive steps were taken until a schedule for the entire year of 2021 was produced. Each step required optimising one of the mixed-integer linear program (MILP) formulations described in Section 6.5.2.1.3, all of which used a single binary variable for each interval in the forecast horizon to prevent the BESS from simultaneously charging and discharging in the same dispatch interval (Shafiee et al., 2016; Wang et al., 2017; Yurdakul and Billimoria, 2023a).

The perfect foresight model and RHOC simulations were implemented in Julia (Bezanson et al., 2017). The MILP formulations were written in the JuMP modelling language (Lubin et al., 2023) and solved using HiGHs (Huangfu and Hall, 2018) with a 1% relative MIP gap tolerance and a 30 second time limit. These solver options were chosen such that solutions of a reasonable quality were attained whilst ensuring that the time required to solve more than 100,000 successive steps for a year-long schedule would not be prohibitive. Figures were generated using Makie.jl (Danisch and Krumbiegel, 2021). All of the aforementioned packages and this study’s source code (Prakash, 2023d) are open-source and freely-available.

8.5.2.1.3 Sensitivity analysis

A sensitivity analysis was conducted to test the impact of information quality (i.e. perfect versus imperfect information) on the annual arbitrage revenues of BESSs with different:

1. Storage durations. These ranged from 15 minutes to 4 hours for a 100 MW BESS. This power capacity and the duration range tested are reflective of those of BESS resources that are currently participating in the NEM;
2. Lookahead horizon lengths. Each step involves optimising the BESS using actual or forecast price data for just the next dispatch interval (5 minutes) or as far as 15 hours (900 minutes) out; and
3. Interpretations of arbitrage opportunities. These are modelled via the optimisation problem formulations described in Table 2.

8.5.2.1.4 Assumptions and limitations

Below, we outline the simplifying assumptions made by the study with regards to BESS dispatch and market participation. We discuss assumptions related to the BESS’s operating characteristics in Appendix 9.1.

• The BESS does not participate in FCAS or provide other system services. Modelling these would likely reduce the power and/or energy capacity available for arbitrage and also require the scheduler to consider FCAS price forecasts.
• We optimise the BESS for dispatch decisions, but market participants would actually optimise their rebids. In theory, appropriately structuring a BESS’s price-quantity bands should mean that it only participates when it is commercially beneficial to do so whilst minimising or altogether avoiding losses in the event of sudden price swings. In practice, many BESS across the NEM appear to be sacrificing some arbitrage upside by pursuing conservative bidding strategies. These entail a BESS bidding a large portion of its capacity into higher price bands to avoid being dispatched if low or moderate prices eventuate, and only rebidding this capacity into lower price bands when participation is perceived to be favourable (Figure 28). Though at least some BESSs no longer have this “all-or-nothing” approach to conservative bidding (compare 2021 and 2023 in Figure 28), there may be instances in which forecasts fail to predict a moderate-to-high price and a BESS only partially captures or even completely misses a significant revenue opportunity. Though additional losses incurred as a result of this assumption cannot easily be addressed by modelling modifications, we choose to adjust raw reductions in annual arbitrage revenue potential by -9% for a 4 hour BESS, -10% for a 1 hour BESS and -16% for a 15 minute BESS to account for them. We describe our reasoning and the methodology used to derive these adjustments in Appendix 11.
• The scheduling decisions made by the 100MW BESS assume that it is a “small device”, i.e. a price-taker. Given the hockey-stick shape of the NEM’s aggregate supply curves, this assumption is likely to hold for lower prices. However, the BESS could play a role in shaping higher prices, e.g. shifting offered quantities to prevent prices from being depressed or to exercise market power. Doing so successfully would not only rely on accurate price forecasts, but also a good understanding of the BESS’s own market power.
• The BESS maintains a dispatch target for the entire interval instead of ramping linearly between its last and next dispatch target – a requirement in the NEM (Australian Energy Market Operator, 2021d).
• BESS dispatch is not restricted by constraints in the NEM’s dispatch engine.
• The market is settled for each dispatch interval, with the BESS being paid or charged the corresponding spot price for the NSW region. In reality, the regional spot price would be adjusted by the marginal loss factor of the BESS (Australian Energy Market Operator, 2012b).

8.5.2.2 Values of perfect lookahead and information

Inspired by metrics for evaluating solutions obtained from optimisation under uncertainty (Roald et al., 2023), we compute two quantities to separate the impact of imperfect information from that of myopic lookaheads and sub-optimal decision-making:

1. Value of perfect lookahead (VPL), which is obtained by expressing the additional annual revenue earned with access to actual price information, rather than forecast prices, in the lookahead horizon for each RHOC simulation step as a percentage of the perfect foresight annual revenue:

\textrm{VPL} = \frac{\textrm{Revenue}_{\textrm{Actual}} - \textrm{Revenue}_{\textrm{Forecast}}}{\textrm{Revenue}_{\textrm{Perfect foresight}}}

2. Value of perfect information (VPI), which is obtained by expressing the additional annual revenue earned with access to actual price information for the entire year as a percentage of the perfect foresight annual revenue:

\textrm{VPI} = \frac{\textrm{Revenue}_{\textrm{Perfect foresight}} - \textrm{Revenue}_{\textrm{Forecast}}}{\textrm{Revenue}_{\textrm{Perfect foresight}}}

If \textrm{VPI} \approx \textrm{VPL}, then information quality accounts for most of the lost revenue potential. Otherwise if \textrm{VPI} > \textrm{VPL}, then other changes, such as increasing the scheduling lookahead, are required alongside better information to recoup lost revenue potential.

8.5.2.3 Results

Using pre-dispatch price forecasts can have a significant impact on BESS annual arbitrage revenues. Though a BESS scheduled with perfect information but a shorter lookahead horizon length of 1 hour can make some minor detrimental decisions (left plot in Figure 29), using price forecasts can lead to a BESS missing a larger number of revenue opportunities and sometimes incurring significant costs as a consequence of charging during an unanticipated price spike (right plot in Figure 29). Over the entire study year, the modelling suggests that using undiscounted price forecasts across the longest lookahead horizon can reduce BESS arbitrage revenue potential by ~15–20% for a 4 hour BESS, ~40–43% for a 1 hour BESS and as much as ~62–64% for a 15 minute BESS (Figure 30). Adjusting for losses that may be avoided by modelling the optimisation of rebid decisions (see Section 6.5.2.1.4), these reductions in BESS annual arbitrage revenue potential become ~6–11% for a 4 hour BESS, ~30–33% for a 1 hour BESS and ~46–48% for a 15 minute BESS.

Despite the magnitude of the aforementioned revenue reductions, our results also suggest that storage operators can reduce the impact of imperfect information to varying degrees by:

1. Scheduling using longer lookahead horizons. As demonstrated by the reduction in the difference between VPI and VPL from a 5 minute to 15 hour lookahead for the 25 MW BESS, a long lookahead horizon is unsurprisingly important to longer duration storage. However, across most of the sensitivities tested in this study, our results suggest that most of this benefit can be captured with a lookahead horizon as short as 4 hours. This is because for undiscounted simulations, forecast quality accounts for most if not all of the lost revenue potential (i.e. \textrm{VPI} \approx \textrm{VPL}) beyond 4 hours. Medium to longer-term operational planning is also important. Cycling constraints, such as those imposed by manufacturer warranties, may incentivise BESS operators to “preserve” cycles for the best opportunities (e.g. May to August in this study, which is, as Figure 31 shows, when the perfect foresight BESS was cycled the most).

2. Discounting price forecasts further into the future when scheduling BESS with sub-hourly storage durations. The exponential function slightly outperforms the hyperbolic function mostly likely due to the latter’s heavier discounting price forecasts closer to real-time, which tend to be more accurate (see Appendix 10). Using the discounting methods tested in this study leads to worse outcomes for the 4 hour BESS as longer-term arbitrage opportunities are devalued (e.g. a cost to charge now is evaluated against a discounted future revenue earned from discharging).

8.6 Options for improving scheduling

The results from Section 6.5.2 suggest that the increasing frequency and severity of price forecast errors in the NEM’s pre-dispatch processes could be hampering their function as a scheduling coordination platform. In particular, flexible resources such as ESRs can lose significant fractions of their potential revenue to instances of sudden and extreme price divergence – products of the NEM’s fast real-time market, its lenient bidding rules, common market participant bidding strategies and inelastic demand. In this section, we discuss options for market participants to mitigate the impact of imperfect information, and changes to centralised knowledge processes and market design that policy-makers could implement to improve resource scheduling outcomes.

Market participants need not solely rely on centralised knowledge processes given that they can generate either their own point forecasts for use in deterministic scheduling methods, or forecasted distributions of prices (or price errors) for use in stochastic or robust scheduling methods (discussed in Section 6.4.2.1.3). However, in markets with blind auctions, producing private forecasts requires using historical data that likely reflect past market dynamics (Lago et al., 2021a). In contrast, pre-dispatch is mechanistic – the best-available forward-looking information, which includes forecasted demand and constraints and the latest set of sealed bids and offers, is used in a modified copy of the market clearing engine (Trebbien et al., 2023). Market participants desiring a balanced solution would ideally use both private and pre-dispatch forecasts in their decision-making, as the former may be more robust to “noisy” market processes whilst the latter better reflects contemporaneous market dynamics and system constraints. Beyond improving price forecast accuracy, market participants should consider two scheduling changes that could improve market revenues. Firstly, scheduling algorithm modifications (e.g. as suggested in Section 6.5.2.3, longer lookaheads and the tested forecast discounting methods for ESRs with short storage durations) may deliver material improvements whilst being simple to implement. Secondly, rather than pursuing all-or-nothing (Section 6.5.2.1.4) or hockey-stick bidding strategies, market participants could appropriately structure their price-quantity bands to reflect costs and manage price risk. However, doing so would require market participants to partially divulge otherwise private information to the broader market and, for ESRs, could lead to increased cycling.

Though there are risks associated with “analytical monocultures” in which the “wisdom of the crowd” is replaced by shared beliefs and thus correlated actions (Bowles et al., 2017; Bronk, 2013), improving the scheduling decision support offered by centralised knowledge processes may be necessary to address existing information and power asymmetries, and could become essential should policy-makers pursue two-sided or hierarchical market architectures that enable smaller consumer-owned energy resources to participate in real-time markets (Hogan, 2019; Kristov et al., 2016). One approach to improving decision support is to provide more information to market participants. This could not only involve publishing additional metrics (e.g. forecast uncertainty measures or the regional aggregate energy availability of energy-constrained resources) (Australian Energy Market Commission, 2023c) but also implementing probabilistic or interactive pre-dispatch engines that provide market participants with a broader perspective of possible market outcomes than the pre-dispatch sensitivities discussed in Section 6.4.1.2. Notwithstanding the transparency benefits, the costs of increased information provision could exceed its benefits if it has little effect on market participant decision-making due to greater complexity (e.g. if pre-dispatch deviates from the real-time market clearing algorithm) or the additional information is not immediately decision-relevant. It could even have deleterious effects should it provide information that reveals opportunities for gaming or collusion to market participants (Creative Energy Consulting, 2020; Von Der Fehr, 2013).

Another approach to improving the decision support offered by centralised knowledge processes is to promote schedule and price convergence. One low-regret option that might assist with convergence is increasing the frequency at which pre-dispatch and PASA are published. More frequent information provision may be particularly effective for a future NEM that will likely consist of many flexible resources that can rapidly respond to changing market conditions. However, this change’s feasibility is limited by the minimum solution times for these processes and, more importantly, it does not reduce the likelihood of sudden and extreme price swings (Section 6.5.1). Addressing these swings may require restricting rebidding since the freedom to readily shift participation preferences frequently, dramatically and within seconds of delivery is enabling the use of all-or-nothing and hockey-stick bidding strategies that contribute to their occurrence. Policy-makers could consider rebid count/frequency restrictions or a “soft” gate closure, which would only allow rebids after a gate closure time close to delivery (e.g. 5 minutes ahead) if there are sudden changes in a resource’s technical status (e.g. forced outage). These sorts of rebidding restrictions might better incentivise market participants to submit a well-structured final rebid that not only reflects their costs and tolerance for price risk, but also hedges against changes from the time of the final rebid to delivery (e.g. demand errors, forced outages). This is also a low-regret option; even if it fails to elicit changes in bidding behaviour, restricted rebidding will at least improve transparency and reduce rebid volumes, thus assisting the market regulator with their assessment of rebid compliance.

Other options could prove to be more effective at improving ESR scheduling outcomes but are challenging to implement as they constitute drastic changes to market design and structure. Several of these options not only increase technical or decision-making complexity, but could also be politically infeasible as they are incongruous with norms in the NEM and the values enshrined in it its rules (e.g. minimising the confiscation of market participant property rights) (Conejo and Sioshansi, 2018):

• Whilst a binding ahead market can provide a degree of schedule and revenue certainty for ESRs, they impose constraints upon flexible systems (Nelson et al., 2018), require mechanisms to reconcile ahead and real-time prices (Ela et al., 2016; Hogan, 2016) and have thus far been rejected in reform processes by both stakeholders and rule-makers in favour of augmenting the NEM’s single platform design (Australian Energy Market Commission, 2023d; Energy Security Board, 2021b).
• Alternative ESR participation models, such as those implemented in US markets (Ela, 2021; Singhal and Ela, 2019), could provide AEMO with a greater degree of control over ESR operation to minimise any detrimental outcomes arising from the market participation choices and preferences discussed in Section 6.4.2.1. However, these participation models would require multi-part bid formats and significant modifications to market participation rules and the NEM dispatch engine to enable centralised resource scheduling and multi-period optimisation, respectively (Billimoria and Simshauser, 2023; Herrero et al., 2020).
• Scheduled demand-side participation could improve price convergence and mitigate hockey-stick pricing, but may require significant structural and market design changes beyond the more incremental market rules that have been adopted (Australian Energy Market Commission, 2020e) or proposed (Australian Energy Market Commission, 2023e).

8.7 Conclusion

With growing deployments of flexible yet potentially energy-constrained VRE and ESRs and increasingly active demand-side resources, policy-makers worldwide are looking towards granular, faster and more flexible electricity markets to effectively and efficiently operate decarbonised power systems. However, achieving good operational outcomes is contingent upon scheduling coordination delivered through sound market participant practices, appropriate market participation rules and purpose-fit knowledge process configurations.

Our work highlights that the increasing frequency and severity of price forecast errors in the NEM’s centralised knowledge processes can lead to sub-optimal scheduling outcomes for BESS ESRs (from ~6–11% reduction in potential annual arbitrage revenue for a 4 hour BESS to almost 50% for a 15 minute BESS) and other scheduled resources that participate in the NEM’s real-time market. Whilst market participants can mitigate the impact of imperfect information by increasing scheduling lookaheads, modifying scheduling algorithms and/or producing robust price forecasts themselves, these changes alone cannot deliver centralised knowledge processes that guarantee “information adequacy” to a diverse range of market participants. This feature is particularly important for market designs and system architectures that aim to enable the participation of consumer-owned energy resources in electricity markets.

While some of the market design options we discussed may prove effective in improving resource scheduling outcomes, they predominantly consist of larger market design or structural changes that would be particularly challenging to implement in the NEM. Instead, we focus on how schedule and price convergence could be improved. One low-regret option is to increase the frequency at which centralised knowledge processes are run. However, just as continuous trading can overwhelm exchanges and induce an inefficient “arms race for speed” (Ahlqvist et al., 2022; Budish et al., 2015; Silva-Rodriguez et al., 2022), our analysis suggests that bidding, when continuous and unrestricted, may have deleterious impacts on system schedule convergence and thus system balancing. As such, we recommend that policy-makers in the NEM consider market participation restrictions that might better incentivise truthful or, at the very least, structured market participant bidding strategies that are less likely to drive these phenomena. Furthermore, our analysis serves as a reminder to policy-makers elsewhere to exercise caution when making the short-term electricity markets in their jurisdictions faster and more flexible.

Data and source code availability

The data used in this study were made publicly available by the Australian Energy Market Operator through their Nemweb portal (Australian Energy Market Operator, 2023c) and were obtained using two open-source tools: NEMOSIS (Gorman et al., 2018) and NEMSEER (Prakash et al., 2023b).

The source code (including data extraction through the aforementioned tools) and results from this study are hosted in two GitHub repositories:

• For material related to the analysis of prices, price forecast errors and the battery energy storage system modelling, please refer to this repository: https://github.com/prakaa/NEMStorageUnderUncertainty (Prakash, 2023d).
• For material related to the analysis of facility numbers and market participant (re)bidding, please refer to this repository: https://github.com/prakaa/nem-rebidding-analysis-2012-2021 (Prakash, 2023e).

9 Conclusion

Prompted by rising deployments of VRE, policy-makers are contemplating redesigning operational practices in their jurisdictions to more effectively and efficiently balance decarbonising power systems and electricity markets. Given the challenges and complexities of the design problem, achieving these outcomes is likely to require implementing flexible, “second-best” and context-specific solutions.

This thesis aims to address the following research question:

Given existing design challenges and increasing penetrations of variable renewable energy resources as energy transition proceeds, how should we design operational practices for balancing electricity markets?

The broad scope of this research question was narrowed through three research objectives that were addressed through modelling and detailed empirical studies of facets of the Australian National Electricity Market. In this chapter, I first summarise the main findings of the research within this thesis according to the research objective they address. Specifically, Section 7.1 summarises insights from Chapter 4 that address Research Objective 1, Section 7.2 summarises the findings from Chapter 5 that address Research Objective 3, and Section 7.3 summarises the findings from Chapter 6 that address Research Objective 3. I then highlight potential opportunities for future work in Section 7.4. Finally, I provide brief closing remarks in Section 7.5.

9.1 Research Objective 1

To determine what features are needed in centrally-coordinated arrangements for procuring frequency control services during energy transition.

Frequency control services are an important component of a system operator’s balancing toolkit and are critical to ensuring that imbalances are quickly addressed. The design of frequency control arrangements in restructured electricity industries will need to be revisited as resource mixes, network topologies and broader market arrangements and policy settings change with growing penetrations of variable renewable energy resources.

Based on the review of North American and European frequency control arrangements and the analysis of the NEM’s presented in Chapter 4, I offer four insights consisting of desirable features and design principles for policy-makers revisiting their jurisdiction’s frequency control arrangements during energy transition:

1. Control deficiencies may not be addressable through introducing new frequency control services. While this solution may address emerging needs, such as low-inertia operation, policy-makers need to better understand the interdependency, interoperability and interchangeability between frequency control services and the interactions with other technical attributes of the power system (e.g. system strength) to ensure that frequency control is first and foremost effective. Once this has been achieved, the short-run efficiency of arrangements can be improved through mechanisms such as dynamic and probabilistic dimensioning and co-optimising the procurement of interchangeable frequency control services.

2. A dynamically efficient outcome in some power systems may require additional investments in capability. Prices for frequency control services can be strengthened through scarcity pricing, which may better reflect the system’s preference for security and reliability. Such pricing mechanisms are complementary to appropriate and efficient cost-allocation based on causation or needs. Both efficient price formation and cost-allocation will improve the potential for frequency control services derivatives, which may assist in providing price signals for investment.

3. System operators should systematically and frequently verify frequency control service delivery, where relevant, and withhold or penalise remuneration when delivery is deemed to be insufficient. If such monitoring is in place, remuneration can be performance-based to drive the provision of high quality frequency control services. Performance monitoring would also enable the system operator to assess frequency control arrangements and identify any deficiencies in control action or procurement.

4. During energy transition, a suitable set of frequency control arrangements will most likely involve a combination of market-based and regulatory mechanisms. Frequency control is a power system public good and achieving frequency stability requires a degree of coordination and cooperation between resources. These characteristics make it difficult to establish complete markets for frequency control services, and an emphasis on market solutions may obscure these characteristics to market participants and undermine effective control. In contrast, regulatory mechanisms may prove to be more robust and resilient in the face of uncertainties, particularly those that are exogenous to the power system (e.g. climate risk). Regardless of whether arrangements are skewed towards market-based mechanisms or regulatory mechanisms, designers should be more forward-looking and avoid assumptions regarding the provision of frequency control capabilities over time, particularly when there is a pervasive competition norm and effective frequency control relies on sequential and hierarchical control actions.

9.2 Research Objective 2

To better understand how balancing flexibility capabilities in scheduling timeframes are changing during energy transition, and how these changes might impact the suitability and design of more decentralised operational balancing practices.

There is a role for empirical studies examining whether decentralised operational balancing practices, such as markets for services and products, are purpose-fit to deliver the balancing flexibility requirements of electricity markets in transition – particularly given that new markets for flexibility (e.g. reserve product markets) can introduce additional costs, constraints and complexity, and even encroach upon the functions of existing operational practices.

By quantifying the time-varying spectrum of balancing flexibility capabilities in scheduling timeframes for historical and projected resource mixes in two regions of the NEM in Chapter 5, I show that with higher penetrations of VRE: 1) downwards flexibility margins can be exhausted around noon if wind and solar are unable or unwilling to provide it, 2) upwards flexibility becomes more scarce during morning and evening peak demand events and 3) a greater portion of upwards flexibility is provided by energy-limited resources. Based on these findings and an assessment of various flexibility design options, I recommend that policy-makers examine how existing operational practices can be augmented to elicit upwards flexibility provision, and that duration specifications and sustained footroom procurement be considered for reserve products.

9.3 Research Objective 3

To explore how more decentralised operational balancing practices can be configured to maximise the deployability of balancing flexibility in scheduling timeframes.

Solely examining capabilities is insufficient for determining the actual availability of balancing flexibility in electricity markets given that the latter is also shaped by market participation decisions. As such, I also examine some aspects of the deployability of balancing flexibility capabilities. In Chapter 6, I explore how market information, participation rules and participant operational strategies affect scheduling decisions and thus balancing flexibility provision from energy storage resources, which are widely touted to be critical to balancing decarbonised power systems. I first highlight the increasing frequency and severity of errors in AEMO-generated price forecasts, which are widely used by market participants in the NEM to schedule their resources, and propose a hypothesis that market participant (re)bidding is partially responsible for this phenomenon. I then model the extent to which BESS wholesale energy market arbitrage revenues might be reduced (compared to perfect foresight operation) should these forecasts guide battery energy storage scheduling. I find that revenue reductions can be significant, ranging from ~6–11% reduction in potential annual arbitrage revenue for a 4 hour BESS to almost 50% for a 15 minute BESS. Based on the findings from these analyses, I discuss potential changes to market participant scheduling strategies and market design that could improve scheduling outcomes. I recommend that Australian policy-makers not only increase the frequency at which centralised knowledge processes are run, but also consider whether stricter market participation restrictions might incentivise participant bidding strategies that are less likely to induce sudden price forecast swings that can hamper effective scheduling.

9.4 Future work

The broad scope of the research question and the various limitations of the work within this thesis mean that there are multiple potential improvements and directions for future work. In the subsections below, I highlight the most significant of these improvements and directions.

9.4.1 Greater consideration of demand response and distributed, consumer-owned energy resources

While Chapters 4, 5 and 6 discussed how demand response and distributed, consumer-owned energy resources have impacted or might impact operational practice design in the NEM to some extent, it is critical for future research is to gain a deeper understanding of how electricity markets can harness the numerous benefits presented by demand response and distributed, consumer-owned energy resources offering balancing flexibility to the system. The financial benefits of doing so in the NEM are estimated to be 1.0–6.3 billion AUD by 2030-2040 (Australian Energy Market Commission, 2023f). There are three avenues for future work in this space that should be pursued:

1. Policy-makers must evaluate whether incremental adjustments to operational practices and market design offer material benefits. This involves not only advancing “enabling” reform, such as the flexible trading agreements (Australian Energy Market Commission, 2023f) and central dispatch visibility and integration mechanisms (Australian Energy Market Commission, 2023g) being considered in the NEM, but also taking a holistic view when assessing seemingly unrelated reform pathways. An example of the latter is, as I discuss in Section 5.3, that technical or market participation requirements for reserve product markets may ultimately hinder balancing flexibility provision from smaller resources.

2. Policy-makers must develop a vision for a target future grid architecture. An important component of this second avenue is determining the extent to which arrangements should be “layered”; that is, whether new distribution system operator roles should be created and what these new entities should be responsible for (Energy Security Board, 2024; Kristov et al., 2016; Schittekatte and Pototschnig, 2022).

3. Research on the needs, preferences and motivations of energy consumers should be continued as arrangements change. Existing work has already challenged the notion that financial incentives alone (e.g. residential and business tariffs) are sufficient to unlock flexibility from demand response and distributed, consumer-owned energy resources (Adams et al., 2021; Ransan-Cooper et al., 2021; Roberts et al., 2023).

9.4.2 Interactions with investment timeframes

The design of practices and policies in operational timeframes should ideally consider those relevant in investment and planning timeframes as they are inextricably linked. For instance, as discussed in Section 5.3, resource adequacy policies implemented alongside the operation of short-term wholesale electricity markets (e.g. longer-term firming revenue or capacity contracts) may weaken the case for building investment signals into or altogether implementing new markets for flexibility.

Holistic design processes should not only work from the “top-down” (i.e. from investment and planning to operational timeframes), but also from the “bottom-up”. With an increasing number of countries adopting hybrid market arrangements (“auctioning of long-term contracts to support investment, together with short-term energy markets”) to drive decarbonisation and satisfy security-of-supply requirements (Keppler et al., 2022; Roques and Finon, 2017), further research on contract design is required to ensure that long-term contracts preserve short-term market scheduling incentives. Neglecting this interdependency can have deleterious effects; for example, the contracts-for-difference issued to VRE resources through reverse auctions in Australia have provided project owners with investment support, but have also diminished real-time market price signals and thus incentives to assist with system balancing seen by project operators (MacGill and Esplin, 2020). A notable piece of work in this area is Billimoria and Simshauser (2023), which investigates how the design of longer-term storage contracts affects the alignment between financial incentives for market participants and the need of the system.

9.4.3 Empirical studies of market participant behaviour

With a specific focus on market information, Chapter 6 discusses behavioural factors and market participant preferences that influence resource scheduling. However, while most of the studies reviewed in Section 6.4.2.1 and the storage scheduling simulations described in Section 6.5.2 model market participant behaviour, there is a need for more in-depth empirical research on market participant behaviour. Analysing market participant rebidding time series data could offer policy-makers with a clearer picture of how participants actually offer balancing flexibility to the system in their jurisdictions. The findings from these studies could be used to inform flexibility assessment modelling efforts (e.g. input bidding assumptions for market models). Moreover, such studies are important to understanding how market participants manage resources across a portfolio – a significant concern in jurisdictions such as the NEM that must contend with supply-side concentration and vertical re-integration (i.e. firms that operate a retail business whilst also owning generation assets) (Australian Energy Regulator, 2022a; Chester, 2024; Simshauser, 2019).

9.4.4 Modelling the network

The system models in Chapter 5 assumed that each modelled market region is a copper-plate network with no interconnection to other regions, and the BESS dispatch models in Chapter 6 assumed that BESS dispatch is not restricted by constraints in the NEM’s dispatch engine. As discussed in Chapter 5, transmission network constraints arising from thermal and stability limits are likely to reduce the balancing flexibility available to the system, whereas interconnection is likely to change the quantity and composition of balancing flexibility under certain conditions. Regardless, modelling both network constraints and interconnection will provide a more accurate picture of the extent to which balancing flexibility capabilities can be realised.

9.5 Closing remarks

This thesis adopts an empirical, “second-best” and context-specific approach to addressing the question of how should we design operational practices for balancing electricity markets with growing penetrations of renewable energy.

Through a review of international frequency control arrangements and an empirical study of arrangements in the Australian NEM, I offer four insights consisting of desirable features and design principles to policy-makers revisiting the centrally-coordinated arrangements for procuring frequency control services in their jurisdictions during energy transition.

Then, following a quantification of the time-varying spectrum of balancing flexibility capabilities in scheduling timeframes for historical and projected resource mixes in two regions of the NEM, I propose that policy-makers examine how existing operational practices can be augmented to elicit upwards flexibility provision, and recommend that reserve products specify response durations and procure sustained footroom.

Finally, I explore how market information, participation rules and participant operational strategies affect the deployability of balancing flexibility from energy storage resources. Based on the findings from an analysis of AEMO-generated price forecasts commonly used for resource scheduling, I discuss potential changes to market participant scheduling strategies and market design that could improve scheduling outcomes. I recommend augmenting centralised knowledge processes and considering stricter market participation rules to discourage destabilising bidding strategies that may be hampering effective resource scheduling.

While there are several important avenues for future work, including greater consideration of demand response, consumer-owned energy resources and the interactions between operational practices and policies in the investment and planning timeframe, these findings will be valuable to policy-makers in the Australian National Electricity Market and other jurisdictions as their power systems progressively decarbonise. Furthermore, this thesis provides a framework for empirical, context-specific and “second-best” approaches to the design problem that is required for effective and efficient outcomes given the complexities and challenges involved in designing operational practices for electricity markets undergoing energy transition.

10 Data and assumptions used in market simulation

10.1 Resource ramp rates

Separate upwards and downwards ramp rates were modelled for most resource types. For hydro generation and reciprocating engines, maximum upwards and downwards ramp rates were sourced from GHD (2018). For other conventional resources (coal-fired generation, Gas-Steam, CCGT and OCGT), ramp rates in each direction were further separated into a market ramp rate, which was used in the PLEXOS market simulation, and an upper ramp rate, which was used to calculate available reserves/footroom. For these resources, the market ramp rate was calculated using the unit ramp rates used most frequently in NEM dispatch⁴⁰ in 2020, and the upper ramp rate was calculated using resources’ assumed maximum ramp rates in AEMO’s 2020 Inputs and Assumptions workbook (for an example of a comparison, see Figure 32) (Australian Energy Market Operator, 2020b). Additional resources in 2025 were assumed to have the same ramp rate characteristics as newer existing resources of the same technology type.

10.2 Unit commitment and cycling constraints

Many existing flexible conventional resources (OCGT, reciprocating engines and hydro generation) submit dispatch inflexibility profiles to AEMO that contain the resource’s time to start up and reach MSL, the MSL itself, the time required at minimum loading and the time taken to shut down (Australian Energy Market Operator, 2021t). The most frequently offered fast start inflexibility profile of a resource in 2020 was obtained using NEMOSIS (Gorman et al., 2018) and used to calculate its start-up rate, minimum up-time, MSL and shutdown rate. The minimum down-time for these resources was chosen to be equal to the minimum up-time.

For the other conventional resources (CCGT, coal-fired generation and Gas-Steam), minimum up-times, minimum down-times and MSLs were obtained from AEMO’s 2020 Inputs and Assumptions workbook (Australian Energy Market Operator, 2020b) and start-up rates were calculated based on hot or warm start times (i.e. depending on the start state of the resource after being offline for its minimum down-time) obtained from GHD (2018) or Aurecon Australasia (2020). The shut-down rates for these resources were calculated based on actual shutdowns, or those of similar technology types, observed in AEMO dispatch data that was obtained using NEMOSIS (Gorman et al., 2018).

BESS were dispatched by PLEXOS’s arbitrage algorithm subject to charging and discharging efficiencies and maximum and minimum state of charge constraints that corresponded to those assumed within AEMO’s 2020 Inputs and Assumptions workbook (Australian Energy Market Operator, 2020b). Given an assumed economic lifetime of 10 years (Australian Energy Market Operator, 2020b) and 3000 cycles (da Silva Lima et al., 2021) for lithium-ion BESS, a constraint of 300 cycles per year was applied to BESS in each scenario.

10.3 Partial and forced outages

Maintenance rates, forced outage rates (partial and full) and the corresponding mean time taken to repair were modelled for all conventional generation and were sourced from AEMO’s 2020 Inputs and Assumptions workbook (Australian Energy Market Operator, 2020b).

10.4 SA synchronous generation requirement

At present, certain combinations of synchronous generators are required to remain online for power system security in SA. Should ahead processes indicate that the synchronous generation expected to be online and dispatched is inadequate to provide sufficient system strength in SA, AEMO will intervene in the market and direct additional synchronous generation online (Gu et al., 2019). The various sufficient combinations of synchronous generation in SA are outlined in Australian Energy Market Operator (2022g), with a decrease in requirements/increase in the allowable asynchronous generation level following the installation of 4 synchronous condensers (completed in 2021). To model these requirements, a must-run condition was imposed on 3 CCGT units and 1 Gas-Steam unit in 2020, and on 2 CCGT units and 1 Gas-Steam unit in the 2025 scenarios. These combinations reflect a subset of the sufficient combinations outlined in Australian Energy Market Operator (2022g).

10.5 Hydro generation monthly energy constraints

Run-of-river hydro generation and pumped hydro storage in NSW were aggregated and modelled as dispatchable generation with monthly energy constraints. These monthly energy constraints correspond to the average monthly inflows for the Snowy scheme (NSW and Australia’s largest hydro scheme) across financial years 2011 to 2018 (obtained from Australian Energy Market Operator (2020b)). Though this model for hydro does not account for the additional generation that could be extracted from pumped storage, the application of monthly energy constraints could be interpreted as modelling one pattern of run-of-river hydro operation and/or enforcing the same reservoir level at the start and end of each month (and thus at the start and end of each year). Explicitly modelling reservoir schemes, inflows for individual hydro generators and pumping opportunities for pumped hydro storage are likely to improve the accuracy of the methodology proposed in this work for systems with significant shares of hydropower capacity.

10.6 Demand and VRE traces

Chronological demand traces at 5-minute resolution were used in the market simulation. For each region, historical operational demand for 2020 at 5-minute resolution was obtained using NEMOSIS (Gorman et al., 2018) and used as the demand trace for the 2020 scenario. AEMO ISP demand traces were available for each 2025 scenario at half-hourly resolution (Australian Energy Market Operator, 2019f); 5-minute resolution demand traces for each 2025 scenario were produced by scaling 5-minute historical operational demand by a corresponding half-hourly scaling factor, which was calculated as the ratio of the ISP scenario’s 2025 demand trace to the ISP scenario’s 2020 demand trace.

Half-hourly chronological solar PV and wind capacity factor traces were obtained from AEMO’s ISP database for each 2020 scenario (Australian Energy Market Operator, 2019g) and for each 2025 scenario (Australian Energy Market Operator, 2020s). Generation traces were obtained by multiplying the capacity factor trace of a resource by its nameplate capacity. Capacities for existing and committed VRE plants were obtained from AEMO’s 2020 Inputs and Assumptions workbook (Australian Energy Market Operator, 2020b) and any additional VRE capacity that was built out in the 2025 scenarios was assigned to AEMO-designated Renewable Energy Zones (for which capacity factor traces are available) based on the ISP’s generation capacity outlook. The half-hourly generation traces for each resource and Renewable Energy Zone in a region were than aggregated and linearly interpolated for use in the 5-minute resolution market simulation.

10.7 Resource market offers

For all scenarios for a given region, one set of four static price-quantity pairs were used to represent each resource’s offer in the market simulation. Except for hydro generation, offers were priced a priori. The type of the resource determined how each band was priced (price bands for each resource type are outlined in Table ¿tbl:resourceoffers?) ⁴¹:

• For wind and solar PV generators, the entire available forecasted energy was offered at the market floor price to ensure preferential dispatch of VRE where possible.
• For baseload conventional resources (coal-fired generation and Gas-Steam), the first band was priced at or close to the market floor price to ensure the resource’s MSL would clear the market. The second band was priced close to the short-run marginal cost (SRMC) of the resource. The SRMC was calculated using the average heat rate, fuel price and variable operating and maintenance cost of each resource type obtained from Australian Energy Market Operator (2020b). The third band was priced at a premium relative to the resource’s SRMC and the fourth band was offered at the market cap price.
• For peaking generation (OCGT and reciprocating engines), the first band was priced close to the SRMC of each resource, which was calculated in the same manner as for baseload conventional resources. The second and third band were offered at a moderate and higher premium relative to the resource’s SRMC, respectively. The fourth band was offered at the market cap price.
• Hydro generation offers were adjusted iteratively to align the proportions of annual generation and average market prices of the NSW 2020 scenario with those calculated from historical data.

Calibration

Resource offer quantities were used to calibrate the 2020 simulation with historical generation patterns in each state. The quantity of energy in each price band was adjusted in an iterative process of offer adjustment and market simulation to ensure that the proportion of annual generation of a particular resource type in the simulated 2020 scenario was similar to the actual proportion of annual generation for that resource type in 2020. The combination of offer quantities that produced the closest proportions were retained and used for each state’s 2020 and 2025 scenarios. The results of the calibration for NSW and SA are outlined in Table ¿tbl:nswcalibration? and Table ¿tbl:sacalibration?, respectively.

11 Mixed integer linear program formulations

11.1 Assumed battery energy storage system operating characteristics

We model a lithium-ion BESS and assume the following with respect to its operating characteristics:

• The BESS is highly flexible; it has no minimum operating levels and can ramp between charging at its maximum power output in one dispatch interval to discharging at its maximum power output in the next (or vice versa).
• The BESS is only cycled between lower and upper state-of-charge limits (fixed at 10% and 90%, respectively). Such limits are often imposed by storage operators to avoid the accelerated degradation that accompanies deep discharging (particularly for lithium-nickel-manganese-cobalt-oxide batteries) (Xu, 2022), and to ensure that market participation obligations can be met by the storage device. Given these state-of-charge constraints and assuming that the BESS is operated at its nominal temperature, current and voltage, we assume that its charging and discharging efficiencies remain constant and fix both at 91% (including inverter losses) (Davies et al., 2019; Yang et al., 2022). Combined, these efficiencies yield an ESR round-trip efficiency of ~83%, which is consistent with values used in similar studies (McPherson et al., 2020; Xu, 2022; Yurdakul and Billimoria, 2023a).
• Self-discharge losses are negligible for a lithium-ion BESS (Bradbury et al., 2014).
• Given that we only model one year of operation, we ignore BESS capacity fade due to cycle and calendar degradation. The latter constitutes an additional opportunity-cost that may modify the attractiveness of certain arbitrage opportunities, particularly if operation over the entire lifetime of the BESS is modelled (Watts and MacGill, 2022; Xu, 2022).

11.2 Nomenclature

11.2.1 Indices and sets

11.2.2 Parameters

11.2.3 Variables

11.3 Arbitrage

Equation ¿eq:arb-1a? maximises arbitrage revenue over the scheduling lookahead horizon. Equation ¿eq:arb-1c? and Equation ¿eq:arb-1d? ensure that discharging and charging, respectively, are greater than or equal to 0 MW, and Equation ¿eq:arb-1b? introduces binary charge state variables that are used in Equation ¿eq:arb-1e? and Equation ¿eq:arb-1f? to enforce the BESS maximum power capacity limit and prevent the BESS from simultaneously discharging and charging in the same dispatch interval. Equation ¿eq:arb-1g? enforces BESS energy storage limits (the rationale for which we previously discussed in Section 9.1), and Equation ¿eq:arb-1h? and Equation ¿eq:arb-1i? are intertemporal constraints that model BESS state-of-charge evolution.

11.4 Arbitrage with throughput penalty

Equation ¿eq:arb-2a? maximises arbitrage revenue over the scheduling lookahead horizon given a penalty on the throughput accrued by the BESS over the optimisation window. The penalty corresponds to the additional BESS throughput accrued over the scheduling lookahead horizon (i.e. d_T - d_0) divided by the assumed BESS warrantied throughput lifetime (d_{\textrm{lifetime}}) and multiplied by the assumed BESS capital cost (the product of e_{\textrm{rated}} and c_{\textrm{capital}}) (see Table 2 for assumed values). Constraints Equation ¿eq:arb-2j? and Equation ¿eq:arb-2k? are intertemporal constraints that model BESS cumulative throughput evolution. All other constraints are described in Section 9.3.

11.5 Arbitrage with discounting

Equation ¿eq:arb-3a? maximises arbitrage revenue over the scheduling lookahead horizon given discounted future prices (i.e. \lambda_t DF(r,t-t_0)) and a penalty on the throughput accrued by the BESS over the optimisation window. The discount function DF is either exponential or hyperbolic, and takes a discount rate r and forecast ahead time t-t_0 as arguments. Refer to the next appendix (Section 10) for the discount function formulae, the methodology for determining discount rates and the discount rate values used in this study’s storage modelling. All other constraints are described in Section 9.3 and Section 9.4.

12 Methodology for discounting future price forecasts

Delay discounting was used to model a scheduler’s belief that price forecasts will improve as the forecast run time approaches the forecasted delivery time. In other words, discounting price forecasts further into the future represents a time preference for information. The rationale for discounting price forecasts is that it can provide robustness to ESR operation by reducing the attractiveness of uncertain opportunities. However it is also problematic as “devaluing” revenues and costs in the near future (i.e. up to a day-ahead) is not reflective of the time periods over which a storage operator’s time preferences are stronger (i.e. over multiple months and years). Devaluing future revenues and costs could also lead to missed opportunities (e.g. due to discounted price spikes) and poor decisions (e.g. discounted lower prices that make charging more attractive than it otherwise would be).

Two discounting functions were tested: an exponential discounting function (Equation 13), which is commonly used in finance and neoclassical economics, and a hyperbolic discounting function (Equation 14), which has been used to model empirical evidence of intertemporal inconsistency in decision-making (Ainslie, 1975; Grüne-Yanoff, 2015). Instead of using the hyperbolic discount function to model the choice of a decision-maker, we use it to model a potential belief about information they might hold: that price forecasts further into the future are likely to be more-or-less equally “untrustworthy” (e.g. a forecast made 8 hours ahead might be as “inaccurate” as a forecast made 12 hours ahead).

DF(r, t-t_0) = e^{-r(t-t_0)}\qquad{(13)}

DF(r, t-t_0) = \frac{1}{1+r(t-t_0)}\qquad{(14)}

Given their importance to market participants, counts of significant pre-dispatch price forecast errors (Figure 33) were used to calculate a discount rate for each function (i.e. r, in units hr^{-1}). Significant price forecast error price counts were max-scaled (i.e. counts at each ahead time were divided by the counts at 24 hours ahead) and then subtracted from one to produce the red curve in Figure 34. The exponential and hyperbolic discount functions were then fitted to this curve using curve fitting tools in the scipy package (McKinney, 2010). The values of r obtained from this process (outlined in Table ¿tbl:discount-rates?) were then used alongside their corresponding discount functions in the arbitrage with discounting MILP formulation (Section 9.5).

Though the hyperbolic discount function obtains a better fit and reflects the intuition that forecasts say 15 hours out and 20 hours out are equally questionable, it discounts price forecasts closer to real-time (\lessapprox 6 hours) to a greater degree than the exponential discount function (Figure 34). As we outline in Section 6.5.2.3, this may, in some cases, lead to poorer arbitrage performance than if an exponential discount function were used.

13 Adjusting for optimising dispatch decisions over rebids

Optimising the modelled BESS for dispatch decisions rather than forming bids and offers (as market participants would do for BESSs in the NEM) leads to lower annual arbitrage revenues because of opportunities missed or losses incurred in the three situations described below:

1. The BESS misses revenue opportunities presented by unanticipated price spikes that it would otherwise be dispatched to capture if its (discharge) capacity were offered into lower price bands. However, as we discuss in Section 6.5.2.1.4, BESSs in the NEM appear to be employing conservative bidding strategies to some extent. This means that they may still be missing opportunities to discharge during periods with moderate-to-high prices.

2. The BESS incurs losses from discharging during a negative price period where it would otherwise not. Though a market participant might choose to discharge the BESS into negative prices (e.g. to reduce its state-of-charge so that it might switch to charging at a later interval priced at the market floor), they could avoid such losses by offering (discharge) capacity into non-negative price bands. The magnitude of the losses incurred by a BESS in this situation is limited by the market price floor (-1000 AUD/MW/hr).

3. The BESS incurs losses from charging during high positive prices where it would otherwise not. Though a market participant might choose to charge the BESS during high positive prices (e.g. to increase the BESS’s state-of-charge so that it can discharge in market intervals with even higher prices), they could avoid such losses by bidding (charge) capacity into negative price bands. The magnitude of the losses incurred by a BESS in this situation may be higher than those incurred from discharging during a negative price period as the modelled market price cap is 15,100 AUD/MW/hr.

13.1 Magnitude of additional losses

Since BESSs participating in the NEM may still miss revenue opportunities because they are employing conservative bidding strategies, we focus on quantifying the impact of the additional losses described above in situations 2 and 3. To do so, we use the modelling methodology described in Section 6.5.2.1 to run a RHOC simulation for each BESS configuration where the BESS is provided with actual price data for the binding period (i.e. next 5 minutes) and forecast price data for the remainder of its lookahead in each RHOC simulation step. We call this the “actual price in binding” (APB) variant, in which the BESS should avoid undesirable losses just as it would if its bids and offers were appropriately structured. By comparing the losses (i.e. under situations 2 and 3) for each BESS configuration in the “standard” simulation with forecast data (i.e. entire RHOC simulation step uses forecast price data, as described in Section 6.5.2.1) and the APB variant, we estimate the degree to which reductions in arbitrage revenue potential calculated from the results of the modelling in Section 6.5.2 should be adjusted to better reflect those that BESSs participating in the NEM might actually experience.

Table 3 shows the maximum losses incurred from discharging (i.e. situation 2) and from charging at higher market prices (i.e. situation 3, with a threshold of 500 AUD/MW/hr chosen based on the “detrimental decisions” shown in Figure 29) across all BESS configurations and lookaheads for the standard simulation and APB variant. The maximum losses from discharging into negative prices are small whilst the maximum losses from charging at higher market prices are significant in the standard simulations (21-36% of annual arbitrage revenue). As such, we hereon only consider avoidable losses from charging at higher market prices. Table 3 also shows that maximum charging losses are significantly reduced but not entirely eliminated by providing the BESS with actual price data for the binding period. Because it is still optimal for the BESS to occasionally charge at high prices to enable greater discharge at even higher prices, modelling simplifications that entirely eliminate charging at higher prices are inappropriate.

To derive adjustments, we calculate the mean and median reductions from the standard simulation to the APB variant in charging losses during intervals with market prices greater than or equal to 500 AUD/MW/hr and take the higher of the two for each storage duration. We calculate reductions by storage durations as shorter duration BESS will charge more frequently, choose the mean and median as aggregation functions so that the adjustments reflect a range of BESS operating practices, and ignore results from the 5 minute lookahead schedules since they are not the result of the intertemporal decision-making that would be common for BESS in the NEM. Table 4 shows the mean and median reductions across all formulation and lookaheads (excluding the 5 minute lookahead) simulated for each BESS storage duration. Based on these results, we choose to apply the following adjustments to the raw reductions in arbitrage revenue potential calculated in Section 6.5.2 to better reflect those that would be seen by a BESS optimising its rebids in the NEM:

• -9% for a 4 hour BESS
• -10% for a 1 hour BESS
• -16% for a 15 minute BESS

It is important to note that the APB variant is inappropriate for modelling the impact of using pre-dispatch price forecasts on arbitrage revenue because it provides the BESS scheduler with perfect foresight for the next market interval. This can lead to annual arbitrage revenue improvements (see Table 5, which shows the improvements of the APB variant on the standard simulation) that exceed the maximum reduction in losses incurred from charging during market intervals with prices greater than or equal to 500 AUD/MW/hr (Table 3).

14 NEMSEER article published in the Journal of Open Source Software

References