The invisibles: tracking solar energy

A few years ago the UK's electricity came almost entirely from a few dozen power stations. Today, solar electricity comes from nearly a million homes, schools, hospitals, supermarkets, factories and fields. How do you track a million invisible electricity creators?

Off

It’s difficult to walk into National Grid’s Electricity System Operator (ESO) Control Room without an overwhelming feeling that you shouldn’t be there. A panorama of blinking lights and live data, dominated by a huge screen makes it feel more like mission control than a head office. But this is National Grid ESO’s nerve centre for balancing the electricity network. A space so critical to the UK’s energy infrastructure that only a select few are allowed inside, the ESO Control Room tracks and manages the flow of supply and demand to make sure everything from your kettle to the lights in Wembley Stadium stay on 24/7, 365 days a year...

At the front of the room is a colourful map of the UK’s electricity transmission system network. And at the bottom of this display is a chart that shows where the electricity comes from. It’s called the generation mix and for years it’s listed traditional energy sources – coal, gas, nuclear, and so on.

Three years ago a new figure was added to the board. Photo-Voltaic energy (or PV) means solar power and it’s one of the new renewable kids on the block. PV turns free energy from the sun into usable electricity and it has the potential to save UK bill payers millions of pounds every year. This new figure is a measure of the amount of PV power being generated. It’s based on analysis and calculations by the University of Sheffield. And it has an extraordinary journey that, fittingly for green energy, starts in the windswept wilds of the Peak District 25 years ago...

Mission control

Alastair Buckley is a Professor in Physics at the University of Sheffield and an orienteerer.

Alastair came to Sheffield as a student in large part because of the city’s position as unofficial UK capital of the sport – a cross between running and map reading – with locations like the Peak District offering spectacular locations for racing. His development of ‘PV_Live’ – a real time service for predicting the UK’s solar output – or outturn as it’s called in industry – has revolutionised energy management. But it started with orienteering…

When Alastair arrived in Sheffield in 2008, solar power was just starting to take off. “In 2010, the government was offering feed-in tariffs to encourage commercial installations, so we set up Sheffield Solar Farm to measure how well different kinds of PV array performed,” Alastair explains while we wend our way through the Farm’s rooftop solar panels in the late summer sunshine.

That farm helped develop an understanding of the variables that go to make up an individual unit’s efficiency and output. But while physicists like Alastair were excitedly getting to grips with a new generation of consumer solar panel technology, on the other side of the country, the emergence of this new energy source presented substantial challenges for the National Grid.

National Grid connects supply with demand for 60 million people. For decades that meant gas, coal, nuclear and oil. But solar changed everything.

The biggest factor with solar power was that it had evolved to become an invisible part of the generation mix.

Jack Barber

Former Senior Energy Forecaster, National Grid

His explanation of the step change solar power presented for a huge organisation like National Grid suggests the challenge wasn’t so much to think big as think small. “We were used to dealing with traditional forms of power generation where the individual generators were all at a large scale – producing at least 100 megawatts each. Solar power works at a much smaller scale. There were thousands of small solar panels installed around the country, but none of them were visible to us. From the National Grid’s point of view, solar power was this volatile, unknown factor that kept disrupting our way of working…”

Managing the transmission system network is about balancing generation and demand in real time. Imbalance affects the frequency of the electric current on the system. And those fluctuations in frequency can have results that range from the inconvenient – such as a television that won’t come on – to, in the case of very rare large frequency changes, shutting off sections of the grid and the power outages that incurs. Without having any visibility of how much solar was being generated at any given time, the ESO had to mitigate by having additional coal and gas power stations on standby.

But whilst the emergence of solar presented many operational challenges, for Alastair, it was an opportunity to capture and play with data. And luckily, the emergence of this network of invisible power generation was taking place at the same time as the 21st century’s digital age was transforming the amount of data we could track, analyse and store.

At this time, solar was generated more or less equally by private homeowners and commercial solar farms. A substantial resource, but also a relatively unpredictable one.

And that’s where Alastair’s interest in orienteering provided inspiration. Orienteering’s a close-knit, competitive community. Much like the people running solar microgenerators from their rooftop, orienteers are fascinated by the data they produce and hungry for access to stats. But orienteers had a way of capturing their data.

The website ‘Attackpoint’ is almost 20 years old, which makes it four years older than Facebook and positively ancient in internet terms. Like Facebook, Attackpoint hinges on community – and that community’s data, centrally – to survive. “One of the things that orienteers do is share their training and racing logs.” Alastair explains. “Attackpoint lets you enter and track the times and distances of your training runs and the split times from your races.”

It struck Alastair that you could do something similar with PV data: a central database where people would be invited to submit data on the combinations of solar panels – also known as ‘arrays’ – being installed up and down the country. Named the Microgen Database, it offered users performance analysis of how their systems compared with others – in exchange for Alastair getting access to users’ data.

A crowd-sourcing data collection tool that quickly built a community and fuelled Alastair’s research into UK PV performance, Microgen pulled in monthly uploaded data from over a thousand solar array systems across the UK and provided several journal papers and conference proceedings on characterising PV performance. “Today Microgen gets talked about as an example of citizen science,” Alastair explains. “But back then I wasn’t especially aware of the term. I just thought it was a good way to collect data...”

A chance encounter

In its early days, the Microgen Database was just a small component of a larger project. Alastair was able to fund it as part of other research projects. But over time it became clear that the data that had been collected was more valuable than the other research that was funding it. “We needed to refocus solely on the database”, Alastair explains. “And that meant aligning the database to an industrial need.”

The clock was ticking, and Alastair knew that if the funding stopped and collection of data halted – even for just a short period – the Microgen database would become useless. Continuity was everything. And the potential value of the database was growing by the day thanks to the dramatic expansion of the solar power industry.

Over 800,000 individual solar PV systems were installed in the UK between 2010 and 2017, capable of generating up to 10 gigawatts of electrical power. That’s roughly a quarter of the typical summer day time peak demand. (The record PV contribution to the UK’s energy mix was around 25% in May in both 2018 and 2019). It was an expansion that took Alastair to London for a workshop at the Institution of Mechanical Engineering in Westminster to discuss its implications. And a fateful meeting...

“The idea was for us to identify any barriers that might prevent Britain’s energy system from having considerably more solar power generation in the future.” Alastair remembers. “I walked in and I took a seat next to this young guy, who introduced himself as being from the energy forecasting team at National Grid ESO. He said something about needing loads of data on solar power generation, and I remember being amazed that they had so little, so I took out my iPad and showed him the Microgen Database.”

“I hadn’t seen so much data in one place before!” says Jack. “My job was all about creating forecasting models to predict supply and demand one, two or three days ahead, but especially for the current day. And forecasting models require data. This was just what I needed.”

“Within minutes we were basically planning to put together what would end up being the PV_Live project,” Alastair says...

That plan would be easy to articulate and difficult to deliver. A near real-time UK PV generation figure.

“The idea was quite simple really,” explains Jamie Taylor, a former MPhys student at the University of Sheffield and now Senior Data Scientist. “We’d mine our Microgen database to get accurate generation data from a sample of PV systems for a given half hour, which is the size of period that National Grid works in. We would then scale that up to give a representative real time PV performance across the country, and feed that figure into the ESO.”

And that was the first challenge. Solar’s extraordinary growth meant Microgen’s data set had become too small to scale up to the million systems that made up the UK solar energy network. And useful though its data was, it wasn’t close to real time.

Training the data

Enter forward thinking smart tech energy co, PassivSystems. They’d already been contributing to the Microgen database for a couple of years, but it was the drive from the ESO to provide much more data that really meant the vision for PV Live could be realised. “Microgen fuelled our research and informed the modelling approach that is used in PV_Live,” explains Jamie. “But to actually operate PV_Live it was crucial we got PassivSystems on board.”

PassivSystems massive dataset enabled Alastair and his team to begin the fundamental science of PV_Live: ‘training’ a statistical model that would upscale their data set to predict outturn for the one million systems deployed on the network.

The output they were looking to define was a function – something they could pass a load of inputs into that would produce a national outturn figure. That function consisted of 30 coefficients – everything from the location, orientation and tilt of the PV system to the intrinsic quality of the installation. Once they were defined, the function was ‘trained’ and then evaluated by by feeding it a combination of solar capacity data and near-live data from PassivSystems.

It’s a description that makes data science sound like running a zoo, but Jamie highlights how ‘datamunging’ messy sets of information is a Sheffield speciality.

It’s equal parts scientific discovery and development of algorithms to model performance.

Jamie Taylor

Former MPhys student at the University of Sheffield

“But the best thing for me is that National Grid ESO are using the outputs of our research in real time. Most research projects, especially in physics, tend to be very low TRL (Technology Readiness Level), but PV_Live as we designed it is now part of National Grid ESO’s day-to-day operations. The ‘as-developed’ solution became the ‘as-implemented’ solution, which is pretty rare.”

“The PV data that PV_Live provides is critical to the management of the transmission system,” says Kevin Tilley, who now helps to embed the results of the project within National Grid ESO, “We use the PV data to help forecast and monitor the national power requirements on the network in real time, and shortly after the first delivery of the data, we were able to improve our forecasts of these flows for the first time in years. Engineers in our planning teams have also been able to predict the regional flows more accurately around the network, enabling safer and more economic operation.”

It’s a sunny day. At time of writing the UK is serving 5.8 GW via solar – which means its serving 17% of GB’s demand. And I haven’t had to go to National Grid ESO’s Control Centre to get that figure. The Sheffield Solar website shares the data that goes to the National Grid. Data based on thousands of people across the UK, a chance meeting in a workshop and an exhausting sport that really tells its participants where they are.

Special thanks to The Wellbeck Estate for photography.