Ex.1 Exercise Notebook — Learning with Volve Field Data —¶

Practical Reservoir Engineering — Data-Driven Simulator Series, Session 1 Exercise (Ex.1)

This notebook follows the structure of DataDrivenSimulator_S1_Exercise.pptx (environment setup → data loading → preprocessing → visualization → feature organization) and records the actual loading and visualization of the Equinor Volve field data (exercise excerpt) stored in Research/References/Volve/.

Data used: Volve_BHP_15_9-F-1C.csv / Volve_BHFP_data.csv / Volve_Production_Data_Processed.csv

Step 1: Environment Setup Check¶

In [1]:
import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

plt.rcParams["font.family"] = "Yu Gothic"
plt.rcParams["axes.unicode_minus"] = False

print("pandas", pd.__version__, "/ numpy", np.__version__, "/ matplotlib", matplotlib.__version__)

DATA_DIR = r"Research/References/Volve"  # local path to the Volve exercise CSVs

pandas 3.0.3 / numpy 2.5.0 / matplotlib 3.11.1

Step 2: Data Loading, Preprocessing, and Checking (Start with One Well)¶

First, load a small file such as Volve_BHP_15_9-F-1C.csv (well 15/9-F-1C only, 746 rows) with pandas.read_csv and get used to the operations.

In [2]:
DATA_DIR = r"Research/References/Volve"  # local path to the Volve exercise CSVs
print(df1.shape)
df1.head()

(746, 5)
Out[2]:
DATEPRD NPD_WELL_BORE_NAME ON_STREAM_HRS AVG_DOWNHOLE_PRESSURE AVG_DOWNHOLE_TEMPERATURE
0 07/04/2014 15/9-F-1 C 0.0 0.000 0.000
1 08/04/2014 15/9-F-1 C 0.0 NaN NaN
2 09/04/2014 15/9-F-1 C 0.0 NaN NaN
3 10/04/2014 15/9-F-1 C 0.0 NaN NaN
4 11/04/2014 15/9-F-1 C 0.0 310.376 96.876
In [3]:
df1["DATEPRD"] = pd.to_datetime(df1["DATEPRD"], dayfirst=True)
print(df1["DATEPRD"].min(), "to", df1["DATEPRD"].max())
print(df1.isna().sum())

2014-04-07 00:00:00 to 2016-04-21 00:00:00
DATEPRD                     0
NPD_WELL_BORE_NAME          0
ON_STREAM_HRS               0
AVG_DOWNHOLE_PRESSURE       3
AVG_DOWNHOLE_TEMPERATURE    3
dtype: int64
In [4]:
fig, ax = plt.subplots(figsize=(10, 4.5))
ax.plot(df1["DATEPRD"], df1["AVG_DOWNHOLE_PRESSURE"], color="#2A78D6", linewidth=1.2)
ax.set_title("Well 15/9-F-1C: Bottomhole Flowing Pressure over Time")
ax.set_xlabel("Date"); ax.set_ylabel("Bottomhole flowing pressure (AVG_DOWNHOLE_PRESSURE)")
ax.grid(alpha=0.3)
fig.tight_layout()
plt.show()
No description has been provided for this image

Observation: Near the final rows around April 2016, the value drops sharply to 0. This is physically impossible and is considered to come from a shut-in (shutdown) or measurement failure of the well. In practice, such anomalous values must be removed or flagged before visualization and feature engineering.

Step 3: Checking Missingness and Trends with Data from All 7 Wells¶

In [5]:
DATA_DIR = r"Research/References/Volve"  # local path to the Volve exercise CSVs
df_all["DATEPRD"] = pd.to_datetime(df_all["DATEPRD"], dayfirst=True)
print(df_all.shape)
print(df_all["NPD_WELL_BORE_NAME"].unique().tolist())

(15634, 5)
['15/9-F-1 C', '15/9-F-11', '15/9-F-12', '15/9-F-14', '15/9-F-15 D', '15/9-F-4', '15/9-F-5']
In [6]:
missing_by_well = df_all.groupby("NPD_WELL_BORE_NAME")["AVG_DOWNHOLE_PRESSURE"].apply(lambda s: s.isna().mean())
n_by_well = df_all.groupby("NPD_WELL_BORE_NAME").size()
summary_missing = pd.DataFrame({"rows": n_by_well, "missing_rate": missing_by_well}).sort_values("missing_rate", ascending=False)
print(summary_missing)
print("\nOverall missing rate:", df_all["AVG_DOWNHOLE_PRESSURE"].isna().mean())

                    rows  missing_rate
NPD_WELL_BORE_NAME                    
15/9-F-5            3306      1.000000
15/9-F-4            3327      1.000000
15/9-F-11           1165      0.005150
15/9-F-1 C           746      0.004021
15/9-F-12           3056      0.001963
15/9-F-14           3056      0.001963
15/9-F-15 D          978      0.000000

Overall missing rate: 0.42561084815146477

Observation (important): The pptx material noted that "the AVG_DOWNHOLE_PRESSURE column is about 42.6% missing overall," but checking the real data shows the missingness is not random: the two wells 15/9-F-4 and 15/9-F-5 (water-injection wells) are 100% missing. The remaining five production wells have almost none (about 0–0.5%). In other words, the substance of the "42.6% missing" is a well-attribute issue — "injection wells have no pressure gauge, so data are absent" — which simple interpolation cannot address. For feature engineering, these two wells must be excluded from the pressure series or replaced with a separate variable such as operating hours.

In [7]:
fig, ax = plt.subplots(figsize=(9, 4.5))
summary_missing["missing_rate"].plot(kind="bar", ax=ax, color="#EB6834")
ax.set_title("By Well: Missing Rate of Bottomhole-Pressure Data")
ax.set_ylabel("Missing rate"); ax.set_xlabel("Well name")
for label in ax.get_xticklabels():
    label.set_rotation(30); label.set_ha("right")
fig.tight_layout()
plt.show()
No description has been provided for this image
In [8]:
fig, ax = plt.subplots(figsize=(10, 5))
colors = ["#2A78D6", "#EB6834", "#1BAF7A", "#EDA100", "#E87BA4", "#008300", "#4A3AA7"]
for i, (well, g) in enumerate(df_all.groupby("NPD_WELL_BORE_NAME")):
    g_valid = g.dropna(subset=["AVG_DOWNHOLE_PRESSURE"])
    ax.plot(g_valid["DATEPRD"], g_valid["AVG_DOWNHOLE_PRESSURE"], label=well,
            color=colors[i % len(colors)], linewidth=1.0, alpha=0.85)
ax.set_title("Bottomhole Flowing Pressure over Time by Well (7 wells overlaid)")
ax.set_xlabel("Date"); ax.set_ylabel("Bottomhole flowing pressure")
ax.legend(fontsize=8, ncol=2); ax.grid(alpha=0.3)
fig.tight_layout()
plt.show()
No description has been provided for this image

Observation: Even after dropna, in some wells the pressure drops to 0 for stretches (e.g., 15/9-F-12 from the second half of 2010 onward, 15/9-F-14 partially). Because 0 appears as a numeric value rather than NaN, isna() alone cannot detect it. In exploratory data analysis (EDA), both "NaN" and "physically impossible 0" must be treated as missing/anomalous values — a lesson unique to working with real data.

Step 4: Loading and Checking Production Data (Field-Wide)¶

Preprocessing pitfall: dates in Volve_Production_Data_Processed.csv use DD/MM/YYYY notation (e.g., 01/09/2007 = September 1, 2007). Reading without dayfirst=True misreads the monthly data as pseudo-daily data and disrupts the time-series plot (in fact, at first the dates were misread this way, producing a distorted graph).

In [9]:
DATA_DIR = r"Research/References/Volve"  # local path to the Volve exercise CSVs
df_prod["Date"] = pd.to_datetime(df_prod["Date"], dayfirst=True)  # dayfirst=True is required
print(df_prod.shape)
print("monotonic:", df_prod["Date"].is_monotonic_increasing)
df_prod.head()

(112, 8)
monotonic: True
Out[9]:
Date p (psia) Np (STB) Gp (SCF) Wp (STB) Gi (SCF) Wi (STB) Rp (SCF/STB)
0 2007-09-01 4780.59 0 0 0.0 0 0 0.0
1 2007-10-01 4780.59 0 0 0.0 0 0 0.0
2 2007-11-01 4780.59 0 0 0.0 0 0 0.0
3 2007-12-01 4780.59 0 0 0.0 0 0 0.0
4 2008-01-01 4780.59 0 0 0.0 0 0 0.0
In [10]:
fig, axes = plt.subplots(2, 1, figsize=(10, 7), sharex=True)
axes[0].plot(df_prod["Date"], df_prod["p (psia)"], color="#2A78D6")
axes[0].set_ylabel("Reservoir pressure p (psia)")
axes[0].set_title("Volve: Reservoir Pressure and Cumulative Production over Time (monthly)")
axes[0].grid(alpha=0.3)

axes[1].plot(df_prod["Date"], df_prod["Np (STB)"], color="#1BAF7A", label="Np (cumulative oil)")
ax2 = axes[1].twinx()
ax2.plot(df_prod["Date"], df_prod["Wp (STB)"], color="#E87BA4", label="Wp (cumulative water)")
axes[1].set_ylabel("Np (STB)", color="#1BAF7A")
ax2.set_ylabel("Wp (STB)", color="#E87BA4")
axes[1].set_xlabel("Date")
axes[1].grid(alpha=0.3)
fig.tight_layout()
plt.show()
No description has been provided for this image

Reading the trend: From the second half of 2007, as production starts, pressure drops rapidly (reservoir drawdown); from around 2010 you can see typical waterflood behavior in which water injection (Wi) recovers and stabilizes the pressure. Cumulative oil Np increases as an S-curve, and from 2013 onward the increase of cumulative water Wp becomes prominent (advancing water = a sign of water breakthrough).

Step 5: Feature Organization (Per-Well Summary Table)¶

In [11]:
feat = df_all.groupby("NPD_WELL_BORE_NAME").agg(
    observations=("DATEPRD", "count"),
    start_date=("DATEPRD", "min"),
    end_date=("DATEPRD", "max"),
    mean_on_stream_hrs=("ON_STREAM_HRS", "mean"),
    mean_bhp=("AVG_DOWNHOLE_PRESSURE", "mean"),
    bhp_missing_rate=("AVG_DOWNHOLE_PRESSURE", lambda s: s.isna().mean()),
)
feat[["mean_on_stream_hrs", "mean_bhp", "bhp_missing_rate"]] = feat[["mean_on_stream_hrs", "mean_bhp", "bhp_missing_rate"]].round(2)
feat
Out[11]:
observations start_date end_date mean_on_stream_hrs mean_bhp bhp_missing_rate
NPD_WELL_BORE_NAME
15/9-F-1 C 746 2014-04-07 2016-04-21 13.38 246.67 0.00
15/9-F-11 1165 2013-07-08 2016-09-17 22.32 233.96 0.01
15/9-F-12 3056 2008-02-12 2016-09-17 21.34 80.73 0.00
15/9-F-14 3056 2008-02-12 2016-09-17 20.54 233.07 0.00
15/9-F-15 D 978 2014-01-12 2016-09-17 18.23 226.03 0.00
15/9-F-4 3327 2007-09-01 2016-12-01 20.24 NaN 1.00
15/9-F-5 3306 2007-09-01 2016-09-18 19.17 NaN 1.00
In [12]:
feat.to_csv("well_feature_table.csv", encoding="utf-8-sig")
print("saved: well_feature_table.csv")

saved: well_feature_table.csv

Summary / Handover to Session 2¶

  • Following the flow of the pptx material (E1–E15), we ran through everything from environment setup (pandas/numpy/matplotlib) to data loading, preprocessing, visualization, and feature organization.
  • Three points learned from the real data, to be reflected in the Session 2 material and in cautions to participants:
    1. Dates in Volve_Production_Data_Processed.csv use DD/MM/YYYY notation; forgetting dayfirst=True collapses the monthly data into pseudo-daily data (noticed after actually running it once and getting it wrong).
    2. Wells 15/9-F-4 and 15/9-F-5 (water-injection wells) are 100% missing for bottomhole-pressure data. The "overall 42.6% missing" is structural missingness due to well attributes, not random.
    3. Missingness appears not only as NaN but also as a physically impossible "0" (well shut-in periods). Since isna() alone cannot detect it, there is value in actually plotting and visually checking during EDA.
  • We created a per-well feature table (well_feature_table.csv). In the Session 2 hands-on surrogate-model building, this can serve as reference for analyzing the distribution of observation counts, operating hours, and bottomhole pressure.