New SLA netcdf format, filter doesn't work

[SusanaMSilva]

Hi.

I'm using your codes to filter SLA data, but it doesn't work with the new SLA data netcdf format provided by Copernicus Marine Service. I already used your codes in a previous work and it worked perfectly with the old netcdf files. Can you help me with this problem?

Thank you

[AntSimi]

Could you share input file and code that you use, in order to be able to reproduce issue?

[SusanaMSilva]

data.zip

The dataset-duacs-rep-global-merged-allsat-phy-l4_1633015461903.nc is the new netcdf file and the dt_global_allsat_phy_l4_19930101_20190101.nc is the old version.
In the code, you just have to change the file name.

Thanks

[AntSimi]

How did you get input data?
When i look with ncdump, i saw variable type in double&float which are not original format.

A quick answer could be to create mask like here :

from netCDF4 import Dataset
from matplotlib import pyplot as plt
from py_eddy_tracker.dataset.grid import RegularGridDataset
import numpy as np

h = Dataset('dataset-duacs-rep-global-merged-allsat-phy-l4_1633015461903.nc')

# File have nan and mask ...
h.variables['sla'][:]

masked_array(
  data=[[[nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         ...,
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan]]],
  mask=False,
  fill_value=1e+20)

grid_name, lon_name, lat_name = 'dataset-duacs-rep-global-merged-allsat-phy-l4_1633015461903.nc', 'longitude', 'latitude'
g = RegularGridDataset(grid_name, lon_name, lat_name)

g.display(ax=plt.subplot(111), name='sla')

g.copy('sla', 'sla_filt1')

# Try filtering raw data ...
g.bessel_high_filter('sla_filt1', 500, order=1)
m =g.display(ax=plt.subplot(111), name='sla_filt1', vmin=-.15, vmax=.15)
plt.colorbar(m)

# Try filtering modified data we create a mask with isnan function
g.copy('sla', 'sla_filt2')
d = g.vars['sla_filt2']
g.vars['sla_filt2'] = np.ma.array(d, mask=np.isnan(d))

g.bessel_high_filter('sla_filt2', 500, order=1)
m =g.display(ax=plt.subplot(111), name='sla_filt2', vmin=-.15, vmax=.15)
plt.colorbar(m)

[SusanaMSilva]

Do you mean the new file? I download it directly from Copernicus Marine Service. I spoke with one person in Copernicus and she said that the netcdf file format is different from the old one.
The old ones I download them with ftp, but it wasn't working this time and with motu i get one file with all the steps, but for filtering we need individual time files.

[AntSimi]

When you look to product user manual, page 33, variables type and other things are different from input file that you shared.
I don't know if you use MOTU to download SLA but differences could come from this tools.

[AntSimi]

Option nan_masking allow to replace mask by isnan method ouput.

from matplotlib import pyplot as plt
from py_eddy_tracker.dataset.grid import RegularGridDataset
grid_name, lon_name, lat_name = 'dataset-duacs-rep-global-merged-allsat-phy-l4_1633015461903.nc', 'longitude', 'latitude'
g = RegularGridDataset(grid_name, lon_name, lat_name, nan_masking=True)
g.bessel_high_filter('sla', 500, order=1)
g.display(ax=plt.subplot(111), name='sla', vmin=-.15, vmax=.15)

[CoriPegliasco]

A comment about the CMEMS netcdf format : when downloading SSH maps with the selection tool for area and time, the unique netcdf to download contain double data (not int), without the _FillValue parameter. CMEMS is investigating the problem.

[SusanaMSilva]

Just tried your solution, the second one and it works. Thank you so much.

The first solution doesn't work, it returns: masked array not defined. I guess it must be a thing that I have to define, but I usually use MATLAB to program and I don't know python language very well.

Again, thank you for your help.