Duncan Irving, EMAE oil and gas practice lead for us technology firm Teradata explains how to grapple with big data in the GCC’s next round of E&P investment
The Exploration and Production (E&P) sector faces daily challenges in integrating sub-surface and operations data with analytics due to phenomenal data volumes, increasing diversity of data, and the requirement to make decisions and take action in ever-decreasing time frames.
There are competing agendas to assimilate fresh data from many sources rapidly, whilst keeping it in a state that will allow it to be revisited many years later; understand the meaning of signals and information within it, in the context of data and knowledge already within the enterprise; capture and store insight to drive operational activities and decision making across the business.
In strategic terms, oil and gas companies typically have an architecture whose components perform the function of filtering and archiving data as it enters the organisation, enabling knowledge to be derived from the data, and driving decision and actions.
Each of the above activities is performed in a tactical and compartmentalized manner in the E&P value chain. Data enters each analytical silo either independently of any overarching process, or simultaneously, with little linkage between insights gained from domain experts.
For example, petrophysicists and engineers may work with the same core log data but in different teams, buildings or even countries, and rarely are their insights brought together.
Need to Warehouse
Typically the E&P workflow uses a broad collection of competing applications in a combination of linear and iterative modes on varying formats and copies of original data. Data governance, whilst exemplary in aspiration and in localised application, is generally poor, and knowledge management over the “Life of Field” fall far behind activities of equivalent economic significance in other industries.
In practice, the uptake of data warehousing has been slow. No E&P operator has succeeded in building an upstream data warehouse. The reasons for failure are a combination of analytical and workflow compartmentalization, massive data volumes, poor understanding of the technological requirements by operators, reciprocated by poor understanding of E&P on the part of the data warehousing vendors.
Article continues on next page …
Solutions
Teradata has developed a range of solutions specific to E&P data warehousing and analytical activities which bring performance and rigour to a workflow dominated by niche applications and interoperability challenges.
The solutions include the explicit abilities to store data and carry out calculations in a proven linearly scalable manner, and integrate all data types at the physical level for computational performance (i.e. not the logical level).
This includes pre-stack, post-stack and streaming microseismic data, interpretation data types, and reservoir models; enable remote/thin data access to drive more involved computational workloads; support robust and scalable multi-user access in a time-based, granular transaction-logging environment.
Whilst each aspect of this is individually unremarkable, no other architecture possesses all of these characteristics. However, we appreciate that this is not enough for the requirements of the E&P workflow.
The applications that make E&P such a technologically advanced area of intersecting computational and scientific disciplines also serve to compartmentalize activities to the detriment of knowledge management, data governance, timely insight and long-term productivity.
By integrating data at the physical and logical levels, we break down these silos. The performance of current applications within E&P is compromised by an unscalable approach to data interaction by the underlying databases, data models and file system architectures employed.
Most client applications are limited in capability by the need to move data constantly through disk and memory in the analytical workflow.
Scalability
Scalability needs to address all of these issues but it is more insightful to view architectural components in terms of hardware requirements.
Applications in the E&P workflow are a hence a functional combination of data visualization and interaction (e.g. reservoir characterization), processor-intensive calculations (e.g. reservoir modelling, seismic imaging), and data-intensive calculations (e.g. seismic processing, history matching, stochastic modelling).
Some application environments have an underlying Logical Data Model (e.g. Schlumberger’s Seabed or Landmark’s EDM) which are fit-for-purpose within their own application ecosystems but offer poor extension to domains outside E&P and rarely enable transfer of data or information across application boundaries.
Placing all data in a ‘massively parallelised’ architecture has the advantage of co-locating data with processing resource. In Teradata’s ‘shared nothing’ architecture, this gives unrivalled analytical scalability.
By decoupling and virtualizing the processing resource, the client need only provide a means of visualizing and interacting with the data in line with the user’s expectation of user experience – and Teradata is clear that this expectation is high in E&P.
Active Data Warehousing
The Teradata analytical ecosystem presents a mixed landscape of data and analytical appliances and the Teradata Active Data Warehouse, all accessible from one place. All interactions are performed through this logical layer. Users and data managers are not exposed to storage, bandwidth and computing resource constraints.
A combination of “light touch” analytics and physical integration of data allows users to ask complex questions “why do I always see this sensor behaviour when this geological signature is close by?”
Previously such a question might have required a fracture map, a flooding map, a borehole log and a WITSML feed to be brought together in PowerPoint or a Cave and then a report written with no clear idea of what data might have changed in the intervening time to the next assessment of the data.
This development brings seismic and subsurface data squarely into the “big data” framework as opposed to merely being a large unwieldy volume of data. This underlying technology is a small but vital component to the broader analytical ecosystem.
We can integrate the subsurface with the business. We can make seismic and sensor data drive insight and, using our data management technologies, ensure that every operational and economic decision is taken with the best view of data and the freshest insights. This is efficient and effective use of big data.
