By Aiman El-Ramly, MBA, CMC, Senior Vice President Strategy, ZE PowerGroup Inc.
For many years, electricity markets throughout North America have been in a nearly constant state of transition. The advent of wholesale competition and power marketing can be traced back to a Federal Energy Regulatory Commission (FERC) initiative in 1987 that grew and was emulated in many jurisdictions. The FERC initiative led, among other things, to the formation of the Western Systems Power Pool (WSPP) and with it many of the data requirements common to current market participants. WSPP membership facilitated trade between franchised utilities creating inter-regional trade efficiencies. The need for standardized product definition and price indexing materialized soon thereafter.
In the early days, pricing was by way of splitting the savings and self-reporting; products traded were essentially a 2 X 2 matrix of peak / off-peak hours for firm / non-firm power. Market transparency was minimal and audit ability almost non-existent, except through the reporting to FERC. It was still very much a handshake and a mark-the-blackboard power trading environment. Furthermore, the telecommunications and information technology industries were still in their infancy. Many of the tools of information exchange for commodity trading were primitive by today’s standards, and almost absurdly inefficient: end-of-day fax bulletins of settlement prices; phone calls to brokers for “real-time” prices; data terminals exclusive only to large-scale corporate trading entities.
In essence, the old paradigm of data flow and management was severely limited by 1) the amount of readily available data, 2) the transmission capabilities for data, and 3) the costs of data storage and maintenance.
Today, the data flow and management paradigm for power markets has radically changed. First, electricity markets are described by a complex of Independent System Operators (ISO) and Regional Transmission Operators (RTO) that span much, but not all, of North America. Most of the west remains, oddly, without an ISO-like operation, but almost the whole of the eastern markets is described by interconnected ISOs. In all, there are 19 transmission operators in Canada and the US. Ontario’s Independent Electricity System Operator (IESO) was one of the first markets to open; following shortly behind the Power Pool of Alberta (AESO) in1996, the California ISO (CAISO) in 1997 and PJM in1998.
As the markets continue to evolve, the total volume of data grows exponentially and the products become both more complex and discrete. Simple standardized, exchange-traded contracts have evolved to include increasingly complex contracts (futures, forwards, options, and swaps), which cover a wide range of commodities and energy-related items: power, gas, coal, transmission rights, emissions, and even weather. As the geographical pricing points in power markets become smaller (note the recent conversion of CAISO and ERCOT markets from zonal to nodal systems), data requirements increase by orders of magnitude. And coming up on the horizon, smart metering initiatives require nodal pricing and load information that may apply to areas as small as an aggregate of thousands of customers, or in some cases a node may describe a single large client account.
One of the companies assisting market participants to manage this kind of critical data, ZE PowerGroup Inc.®, based in Vancouver BC, has built a software platform that’s tied into many of the critical markets on the continent. Its ZEMA® data management and analysis suite is designed to capture, store, report on and distribute the mass of data available in the market. The company, focused on consulting and software development, helps a range of market participants by providing electronic tools that can be pivotal when integrated with strategic decision making at the highest levels.
While the IESO may not be the most complex market in North America, it must surely share the honour. The IESO publishes a variety of reports ranging from real time data on Pre-dispatch and Dispatched Market Clearing Prices (MCPs) to hourly and daily data on Transmission Rights, Outages, Trade flows between zones, Settlement Charges, Demand and Supply for regions within IESO control areas, and many others. As of this month, the database housed at ZE PowerGroup contained over 200 million individual data records pertaining to the IESO alone. Everyday ZEMA, through its Data Manager® automatically schedules and processes over 450,000 additional records to its database from the IESO.
Market participants operating in Ontario, therefore, could need to analyze and manage responses to perhaps half a million data points a day. If the participant is operating in multiple regional markets, the data needs grow significantly. If the market participant is also dealing with multiple commodities (power, gas, crude, etc.) then the needs grow again to include data originating from exchanges, brokers, government agencies, counterparties, financial institutions and other reporting entities. Many find themselves collecting millions of data points every day.
Accounting for all the transmission operators, exchanges, brokers, government agencies and so forth in the North American energy industry alone, ZE PowerGroup adds almost 10 million records daily. One can imagine the annual growth. Fortunately, the growth in data requirements has coincided with exponential improvements in data transmission and storage abilities, to the point where hardware is no longer the limiting constraint. Where a significant gap still exists for many market participants however, is the lack of systems to intelligently harness the power of today’s advanced telecommunications infrastructure to capture and analyze the mountains of readily available market data to facilitate strategic decision-making and increased efficiency (physical and economic) in the power markets.
Managing systems to be able to effectively collect millions of data points a day is a highly technical task in itself. However, the issue beyond simple data collection is to determine how corporations analyze and extract intelligence from the data - what are the correlations, trends, spark spreads, heat rates, volatilities, forward curves, mark to market, value at risk … the data combinations are boundless. Consequently, a streamlined data management paradigm is vital to the success of any energy market participant.
It is easy to see how data characteristics begin to describe the necessary operating conditions. Not only that, but data handling and business intelligence capability can be a crucial lynchpin in competitive positioning for many companies. Notwithstanding the corporate need for trading, risk management, settlement, scheduling, accounting and reporting, there are also regulatory requirements for data handling built into SEC’s Sarbanes-Oxley Act, FERC’s adoption of the FASB Asset Retirement Standard, and NERC Cyber Compliance Standards, just to name three. We can quickly see how data governance is mission critical. It is no wonder that the offices of the CFO and CIO are so frequently taking direct interest in managing commodity data.
Market Data -> Metadata -> Accessible/Useful Data
There are several complexities associated with collecting and managing data, particularly when doing so from a multitude of sources. First, the variety of data of interest to market participants ranges from capacity and load reports, to prices, to data without a common unit of measurement like outages. Second, the format of data varies widely, not just between ISOs, but between all the myriad of public and private data vendors. Data can be sourced from multiple locations (FTP, HTTP, HTTPS, email, local disk, etc.) in multiple formats (html, Excel, text, PDF, etc.).
To effectively manage this wide variety of data, ZE PowerGroup has chosen to configure its data parsing technology so as to catalog report types based on individual characteristics of each observation; each report’s units and product types will be assigned into a specific and logical category applicable to the specific commodity. The source data is then organized into a logical structure that can be referenced by generic analytical tools and via display output. ZE PowerGroup leverages its proprietary “metadata” architecture to accomplish this task. Metadata systems describe and apply the necessary attributes that allow for the intelligent cataloging and referencing of data. Without the Metadata layer, the hundreds of millions of data points that a corporation must maintain can become inordinately difficult.
The data management task is an ongoing function. Market structures are constantly in flux and new products and initiatives are always being introduced. Preparing for the release of the California MRTU market, for example, was almost a two year process.
Market participants need a range of resources to support their market activities in and beyond the front, mid and back offices. Data managers and support people to manage systems are necessary to success in this environment. Standardized and flexible technology is essential to ensure that the interfaces between with the entire enterprise energy trade and risk management (ETRM) are seamless and fully effective. The greatest challenge that ZE PowerGroup is called to solve for its clients is to resolve the linkage and supply of market data with business systems.
Build, Buy, or Rent?
The key question for most market participants will be to build, buy or rent their market data systems. The answer to the question requires consideration of the scope of operation, available budget, available skill and risk tolerance. Unfortunately, forgoing data management is not an option, and every day of delay adds cost because the amount of data and products grows progressively. The first step in the build or buy decision process is to specify a clearly defined scope of market and business operation. Next, the data needs and the internal data trail need to be mapped. Most companies are surprised by the depth of data they use and how widespread the use can be. Often the same data is being collected manually or through ad hoc processes across several departments without a systematic understanding or ownership by the corporation. The scoping will allow the corporation to assess the criticality and needs of the users, while providing greater transparency around data flow and internal processes.
Thereafter the corporation can cost the solution and determine whether it has the near term resources and technology to build, and the long term resources to maintain- recognizing it is never as easy as it seems. Integration of technical requirements and business need is often a cliffhanger, taxing corporate resolve. When building complex systems there is an extended time to market, and the effort must be sustained and managed. Significant resources must be dedicated and cannot be allowed to lose focus. Maintenance costs and resources must be allocated annually.
ZE PowerGroup has been building its software for ten years and has logged something like one hundred person years in the development. The ZEMA software suite is continuously adjusted to new market demands. Outsourcing can be attractive given that the required level of sustained effort can be impossible for a small company, while large companies may choose to focus their efforts on their core competencies. ZE PowerGroup has found that most of its happy clients are those who attempted or considered building a solution on their own and discovered that an out-of-the-box flexible solution is a much cheaper and less risky option. The obvious caveat is: as long as all defined needs can be met.