Using the Data Explorer Library

Installation

First, you will need to install it. First clone it from here:

https://gitlab.tudelft.nl/steelelab/data-explorer

You can install it by going to the location where you cloned it to and then run pip install:

pip install .

You will need some libraries. I have not yet figured out how to build this into the pip file. But you will need these:

numpy
xarray
matplotlib
bokeh

With those, you should be ready to go!

Importing the module

The data explorer module is named dexplore and a quasi-convention is to import it as dx:

import dexplore as dx

Loading BokehJS ...

It should trigger a loading of Bokeh. If your bokeh plots become unresponsive (I have had this with browser tabs connected to the jupyterhub), then you can reload the browser tab (not the kernel) and re-run the import to reload the Bokeh javasript into your browser.

Loading data into an xarray

This works for now only for files with meta.txt files that specify the coordinate axis sizes, range and names.

If you do not have meta.txt files, or if your data is in another format, you will need your own code to parse your data into xarrarys:

https://docs.xarray.dev/en/stable/

Here, we will load a random ASCII dat file from the NAS.

dat = "/home/jovyan/steelelab/measurement_data/Triton/Yildiz/"
dat += "data_analysis/data/Y1_2017_07_10_17.24.08_RF_vs_Vg_at0p015K_part2/"
dat += "Y1_2017_07_10_17.24.08_RF_vs_Vg_at0p015K_part2.dat"
data = dx.load_dat_meta_txt(dat)

This is what the resulting object looks like:

data

<xarray.Dataset>
Dimensions:                  (Frequency (Hz): 2001, Voltage (V): 161, Nothing: 1)
Coordinates:
  * Frequency (Hz)           (Frequency (Hz)) float64 3e+07 3.014e+07 ... 3e+08
  * Voltage (V)              (Voltage (V)) float64 4.0 3.75 3.5 ... -35.75 -36.0
  * Nothing                  (Nothing) float64 0.0
Data variables:
    Frequency (Hz) (column)  (Frequency (Hz), Voltage (V), Nothing) float64 3...
    S21re ()                 (Frequency (Hz), Voltage (V), Nothing) float64 -...
    S21im ()                 (Frequency (Hz), Voltage (V), Nothing) float64 -...
    S21dB (dB)               (Frequency (Hz), Voltage (V), Nothing) float64 -...
    S21Ph (rad)              (Frequency (Hz), Voltage (V), Nothing) float64 -...
    Power (dBm)              (Frequency (Hz), Voltage (V), Nothing) float64 -...
    Temperature (K)          (Frequency (Hz), Voltage (V), Nothing) float64 0...
    Voltage (V) (column)     (Frequency (Hz), Voltage (V), Nothing) float64 -...
Attributes:
    datafile:  /home/jovyan/steelelab/measurement_data/Triton/Yildiz/data_ana...
    metafile:  /home/jovyan/steelelab/measurement_data/Triton/Yildiz/data_ana...

xarray.Dataset

• Dimensions:
  • Frequency (Hz): 2001
  • Voltage (V): 161
  • Nothing: 1
• Coordinates: (3)
  • Frequency (Hz)
    (Frequency (Hz))
    float64
    3e+07 3.014e+07 ... 2.999e+08 3e+08

    array([3.00000e+07, 3.01350e+07, 3.02700e+07, ..., 2.99730e+08, 2.99865e+08,
           3.00000e+08])

  • Voltage (V)
    (Voltage (V))
    float64
    4.0 3.75 3.5 ... -35.5 -35.75 -36.0

    array([  4.  ,   3.75,   3.5 ,   3.25,   3.  ,   2.75,   2.5 ,   2.25,   2.  ,
             1.75,   1.5 ,   1.25,   1.  ,   0.75,   0.5 ,   0.25,   0.  ,  -0.25,
            -0.5 ,  -0.75,  -1.  ,  -1.25,  -1.5 ,  -1.75,  -2.  ,  -2.25,  -2.5 ,
            -2.75,  -3.  ,  -3.25,  -3.5 ,  -3.75,  -4.  ,  -4.25,  -4.5 ,  -4.75,
            -5.  ,  -5.25,  -5.5 ,  -5.75,  -6.  ,  -6.25,  -6.5 ,  -6.75,  -7.  ,
            -7.25,  -7.5 ,  -7.75,  -8.  ,  -8.25,  -8.5 ,  -8.75,  -9.  ,  -9.25,
            -9.5 ,  -9.75, -10.  , -10.25, -10.5 , -10.75, -11.  , -11.25, -11.5 ,
           -11.75, -12.  , -12.25, -12.5 , -12.75, -13.  , -13.25, -13.5 , -13.75,
           -14.  , -14.25, -14.5 , -14.75, -15.  , -15.25, -15.5 , -15.75, -16.  ,
           -16.25, -16.5 , -16.75, -17.  , -17.25, -17.5 , -17.75, -18.  , -18.25,
           -18.5 , -18.75, -19.  , -19.25, -19.5 , -19.75, -20.  , -20.25, -20.5 ,
           -20.75, -21.  , -21.25, -21.5 , -21.75, -22.  , -22.25, -22.5 , -22.75,
           -23.  , -23.25, -23.5 , -23.75, -24.  , -24.25, -24.5 , -24.75, -25.  ,
           -25.25, -25.5 , -25.75, -26.  , -26.25, -26.5 , -26.75, -27.  , -27.25,
           -27.5 , -27.75, -28.  , -28.25, -28.5 , -28.75, -29.  , -29.25, -29.5 ,
           -29.75, -30.  , -30.25, -30.5 , -30.75, -31.  , -31.25, -31.5 , -31.75,
           -32.  , -32.25, -32.5 , -32.75, -33.  , -33.25, -33.5 , -33.75, -34.  ,
           -34.25, -34.5 , -34.75, -35.  , -35.25, -35.5 , -35.75, -36.  ])

  • Nothing
    (Nothing)
    float64
    0.0

    array([0.])

• Data variables: (8)
  • Frequency (Hz) (column)
    (Frequency (Hz), Voltage (V), Nothing)
    float64
    3e+07 3e+07 3e+07 ... 3e+08 3e+08

    array([[[3.00000e+07],
            [3.00000e+07],
            [3.00000e+07],
            ...,
            [3.00000e+07],
            [3.00000e+07],
            [3.00000e+07]],
    
           [[3.01350e+07],
            [3.01350e+07],
            [3.01350e+07],
            ...,
            [3.01350e+07],
            [3.01350e+07],
            [3.01350e+07]],
    
           [[3.02700e+07],
            [3.02700e+07],
            [3.02700e+07],
            ...,
    ...
            ...,
            [2.99730e+08],
            [2.99730e+08],
            [2.99730e+08]],
    
           [[2.99865e+08],
            [2.99865e+08],
            [2.99865e+08],
            ...,
            [2.99865e+08],
            [2.99865e+08],
            [2.99865e+08]],
    
           [[3.00000e+08],
            [3.00000e+08],
            [3.00000e+08],
            ...,
            [3.00000e+08],
            [3.00000e+08],
            [3.00000e+08]]])

  • S21re ()
    (Frequency (Hz), Voltage (V), Nothing)
    float64
    -0.008425 -0.00838 ... 0.001969

    array([[[-0.00842528],
            [-0.00838026],
            [-0.00800526],
            ...,
            [-0.00743075],
            [-0.00677491],
            [-0.00452096]],
    
           [[-0.008863  ],
            [-0.00833247],
            [-0.00875748],
            ...,
            [-0.00763287],
            [-0.00659452],
            [-0.0044043 ]],
    
           [[-0.00875554],
            [-0.00891064],
            [-0.00885055],
            ...,
    ...
            ...,
            [ 0.00052085],
            [ 0.00125827],
            [ 0.00127538]],
    
           [[ 0.00462996],
            [ 0.00463751],
            [ 0.00462921],
            ...,
            [ 0.00084823],
            [ 0.00148135],
            [ 0.00136371]],
    
           [[ 0.00477183],
            [ 0.00489952],
            [ 0.00477849],
            ...,
            [ 0.00120769],
            [ 0.00193648],
            [ 0.00196899]]])

  • S21im ()
    (Frequency (Hz), Voltage (V), Nothing)
    float64
    -0.005434 -0.005032 ... 0.006322

    array([[[-0.00543446],
            [-0.00503241],
            [-0.00539019],
            ...,
            [ 0.00030786],
            [ 0.00046963],
            [ 0.00096816]],
    
           [[-0.00451178],
            [-0.00454943],
            [-0.00442467],
            ...,
            [ 0.00097201],
            [ 0.00098332],
            [ 0.00129954]],
    
           [[-0.0035232 ],
            [-0.00364687],
            [-0.00426076],
            ...,
    ...
            ...,
            [ 0.00663169],
            [ 0.00705266],
            [ 0.00650182]],
    
           [[ 0.00442409],
            [ 0.00455917],
            [ 0.00429364],
            ...,
            [ 0.00654509],
            [ 0.00681868],
            [ 0.00641379]],
    
           [[ 0.00407758],
            [ 0.00430837],
            [ 0.00429571],
            ...,
            [ 0.0065649 ],
            [ 0.0068844 ],
            [ 0.00632225]]])

  • S21dB (dB)
    (Frequency (Hz), Voltage (V), Nothing)
    float64
    -39.98 -40.2 ... -42.91 -43.58

    array([[[-39.97753216],
            [-40.19751409],
            [-40.3087185 ],
            ...,
            [-42.57189859],
            [-43.36111483],
            [-46.70065014]],
    
           [[-40.04764193],
            [-40.45143377],
            [-40.16504387],
            ...,
            [-42.27638265],
            [-43.52082927],
            [-46.75992961]],
    
           [[-40.50257164],
            [-40.32924389],
            [-40.1553402 ],
            ...,
    ...
            ...,
            [-43.54080769],
            [-42.89685859],
            [-43.57532471]],
    
           [[-43.87120223],
            [-43.73736239],
            [-43.99408328],
            ...,
            [-43.60934876],
            [-43.12571209],
            [-43.66567335]],
    
           [[-44.04536474],
            [-43.70923103],
            [-43.84185906],
            ...,
            [-43.51089291],
            [-42.91197461],
            [-43.58051943]]])

  • S21Ph (rad)
    (Frequency (Hz), Voltage (V), Nothing)
    float64
    -2.569 -2.601 ... -99.23 -99.26

    array([[[  -2.56872741],
            [  -2.60079972],
            [  -2.54899044],
            ...,
            [   3.1001853 ],
            [   3.07238436],
            [   2.93063005]],
    
           [[  -2.67072498],
            [  -2.64183448],
            [  -2.67375835],
            ...,
            [   3.01492944],
            [   2.99357172],
            [   2.85467346]],
    
           [[  -2.75902194],
            [  -2.75311913],
            [  -2.69292585],
            ...,
    ...
            ...,
            [ -99.03854675],
            [ -99.13672127],
            [ -99.15386697]],
    
           [[-106.05148593],
            [-106.03727003],
            [-106.06634192],
            ...,
            [ -99.08904887],
            [ -99.17409398],
            [ -99.16967078]],
    
           [[-106.10704404],
            [-106.09286475],
            [-106.08190554],
            ...,
            [ -99.14209674],
            [ -99.23436837],
            [ -99.26208592]]])

  • Power (dBm)
    (Frequency (Hz), Voltage (V), Nothing)
    float64
    -20.0 -20.0 -20.0 ... -20.0 -20.0

    array([[[-20.],
            [-20.],
            [-20.],
            ...,
            [-20.],
            [-20.],
            [-20.]],
    
           [[-20.],
            [-20.],
            [-20.],
            ...,
            [-20.],
            [-20.],
            [-20.]],
    
           [[-20.],
            [-20.],
            [-20.],
            ...,
    ...
            ...,
            [-20.],
            [-20.],
            [-20.]],
    
           [[-20.],
            [-20.],
            [-20.],
            ...,
            [-20.],
            [-20.],
            [-20.]],
    
           [[-20.],
            [-20.],
            [-20.],
            ...,
            [-20.],
            [-20.],
            [-20.]]])

  • Temperature (K)
    (Frequency (Hz), Voltage (V), Nothing)
    float64
    0.01555 0.01555 ... 0.01527 0.01525

    array([[[0.0155462],
            [0.0155462],
            [0.0155487],
            ...,
            [0.0152659],
            [0.0152659],
            [0.0152513]],
    
           [[0.0155462],
            [0.0155462],
            [0.0155487],
            ...,
            [0.0152659],
            [0.0152659],
            [0.0152513]],
    
           [[0.0155462],
            [0.0155462],
            [0.0155487],
            ...,
    ...
            ...,
            [0.0152659],
            [0.0152659],
            [0.0152513]],
    
           [[0.0155462],
            [0.0155462],
            [0.0155487],
            ...,
            [0.0152659],
            [0.0152659],
            [0.0152513]],
    
           [[0.0155462],
            [0.0155462],
            [0.0155487],
            ...,
            [0.0152659],
            [0.0152659],
            [0.0152513]]])

  • Voltage (V) (column)
    (Frequency (Hz), Voltage (V), Nothing)
    float64
    -36.0 -35.75 -35.5 ... 3.5 3.75 4.0

    array([[[-36.     ],
            [-35.7501 ],
            [-35.5001 ],
            ...,
            [  3.49999],
            [  3.74999],
            [  3.99999]],
    
           [[-36.     ],
            [-35.7501 ],
            [-35.5001 ],
            ...,
            [  3.49999],
            [  3.74999],
            [  3.99999]],
    
           [[-36.     ],
            [-35.7501 ],
            [-35.5001 ],
            ...,
    ...
            ...,
            [  3.49999],
            [  3.74999],
            [  3.99999]],
    
           [[-36.     ],
            [-35.7501 ],
            [-35.5001 ],
            ...,
            [  3.49999],
            [  3.74999],
            [  3.99999]],
    
           [[-36.     ],
            [-35.7501 ],
            [-35.5001 ],
            ...,
            [  3.49999],
            [  3.74999],
            [  3.99999]]])

• Attributes: (2)

  datafile :

  /home/jovyan/steelelab/measurement_data/Triton/Yildiz/data_analysis/data/Y1_2017_07_10_17.24.08_RF_vs_Vg_at0p015K_part2/Y1_2017_07_10_17.24.08_RF_vs_Vg_at0p015K_part2.dat

  metafile :

  /home/jovyan/steelelab/measurement_data/Triton/Yildiz/data_analysis/data/Y1_2017_07_10_17.24.08_RF_vs_Vg_at0p015K_part2/Y1_2017_07_10_17.24.08_RF_vs_Vg_at0p015K_part2.meta.txt

You can read more about what it can do in the help:

help(dx.load_dat_meta_txt)

Help on function load_dat_meta_txt in module dexplore.load_dat_meta_txt:

load_dat_meta_txt(datfile, delimiter=',')
    Load .dat file with accomanying metadata in meta.txt format specification. 
    
    Parameters
    ----------
    datfile:
        Absolute path to a .dat file.
    delimiter:
        Specify the delimiter. Default is ",".
    
    Returns
    -------
    data:
        xarray containing the data and metadata
    
    This routine expects that the dat file is accompanied by a file with same 
    name but with .dat replaced with .meta.txt, specifying the coordinate and dataset
    names contained in the file according to the spyview specification:
    
    https://nsweb.tn.tudelft.nl/~gsteele/spyview/#meta
    
    Specifically, the meta.txt file format is defined by: 
    
    # Lines that start with "#" are comments, and are ignored
    #
    # The first four lines contain the number of x points (ie. the inner loop, "sweep")
    # followed by the xmin, the xmax, and the xaxis label. 
    101
    -500.0
    400.0
    Sweep gate (mV)
    # The next is the same for y (ie. secondmost inner loop, "loop1")  
    201
    -10.0
    10.0
    Bias (mV)
    # The last is for z (outermost loop, "loop2")
    # If your dataset is only a 2D sweep, you can set these to "1", "0", "1" and "Nothing". 
    11
    0
    9
    Magnetic Field (T)
    # Next, you can optionally add labels for each of the colums in the datset. 
    # If you do not give a label, the data will be given a default
    # data name of "Column X"
    # For each column label, include one line with the column number, and a second
    # line giving the column data label 
    2 
    Current (pA)
    5 
    Time per pixel (ms)

Plot a linecut

This will make a static, inline matplotlib plot. This is handy for saving / exporting as HTML, since the Bokeh plots do not export if you save the notebook as HTML_Embed (or at least, I haven't figured it out). I have configured it to plot the third column by default, since this is for us the most common.

It plot the first line along first axis by default. It will get the axis labels from the data array metadata.

dx.plot_line_cut(data)

You can see more about how to use it here:

help(dx.plot_line_cut)

Help on function plot_line_cut in module dexplore.static_plots:

plot_line_cut(data, dataset=3, axis=0, pos=(0, 0, 0))
    Plot a line cut from the supplied xarray
    
    Parameters
    ----------
    data:
        A xarray which contains (max) 3 coordinates
    dataset: (default 1)
        Which dataset of the array to use for the plot. Can 
        be either and integer corresponding to the index (column
        number for dat files) of the dataset, or it can be a string
        specifying the name of the dataset. 
    axis: (default 0)
        An integer specifying along which coordinate axis the
        linecut should be taken.
    position: (default (0,0,0))
        A tuple of integers specifying the index of the coordinates 
        of the starting point of the linecut. The value of the axis 
        corresonding to the linecut direction coordinate axis
        is ignored.

Interactive linecuts

This will allow you to use controls in the notebook to change the index and axis of the linecut and change which dataset to plot. It will produce a bokeh plot, which means you can interactively pan / zoom through your data.

dx.interactive_linecut(data)

VBox(children=(HBox(children=(Dropdown(description='Linecut Along:', options=('Frequency (Hz)', 'Voltage (V)',…

Plot an colormap

This function will plot a static colormap of your data:

dx.plot_colormap(data)

It takes several optional parameters to change things, such as the colormap name and the gamma normalisation:

help(dx.plot_colormap)

Help on function plot_colormap in module dexplore.static_plots:

plot_colormap(data, dataset=3, cut_axis=2, pos=0, cmap='RdBu_r', gamma=1)
    Plot a 2D colormap from the supplied xarray
    
    Parameters
    ----------
    data:
        A xarray which contains (max) 3 coordinates
    dataset: (default 1)
        Which dataset of the array to use for the plot. Can 
        be either and integer corresponding to the index (column
        number for dat files) of the dataset, or it can be a string
        specifying the name of the dataset. 
    cut_axis: (default 2)
        An integer specifying from which coordinate axis the
        2D colorplot should be sliced.
    position: (default 0)
        Index of plane to be sliced for the 2D colormap.
    cmap: (default "RdBu_r")
        The matplotlib colormap to be used. 
    gamma: (default 1)
        The gamma that should be used for the normalisation. 
        See matplotlib PowerNorm for more details.

Interactive colormap

This will create a bokeh interactive colormap with a slider for changing the gamma, a selector box for choosing the data variable to plot, a dropdown box for picking the colormap, and a slider for clicking through colormaps with the left and right keys.

Right now, the interactive version only supports colormaps of the data along the first two axes of your dataset, since that is what we need in the lab 99.99% of the time. In principle it could be extended to other slices of data with 3 coordinate axes (like plot_colormap() above) but for now, I've kept the code simple.

dx.interactive_colormap(data)

VBox(children=(HBox(children=(FloatSlider(value=1.0, description='gamma:', max=5.0, min=0.05, step=0.01), Drop…

You can find out more about the options here:

help(dx.interactive_colormap)

Help on function interactive_colormap in module dexplore.interactive_colormap:

interactive_colormap(data, initial_var=3, throttle_delay=0.2)
    Generate an interactive colormap from an xarray dataset
    
    Parameters
    ----------
    data:
        A xarray dataset which contains (max) 3 coordinates
    initial_var: (default 3)
        Which dataset variable to start with when opening the widget.
        Either a number index of the dataset variable, or a dataset name. 
    throttle_delay: (default 0.2)
        Throttle time in seconds. To prevent the UI interactions from 
        falling behind, this function throttles calls to the bokeh updates. 
        If your plots are taking a long time to finish updating after 
        moving the slider, you can increase this time.

Combined interactive colormap and linecut

This is a function that generates a combined interactive colorplot and linecut. The position and direction of the linecut can be changed, and the position of the linecut is indicated by a line in the colormap.

For now, linecut position is controlled by a slider. At some point, I will look up how to get this from bokeh click events in the plot: should be easy, just haven't had time yet.

dx.interactive_linecut_and_colormap(data)

VBox(children=(HBox(children=(FloatSlider(value=1.0, description='gamma:', max=5.0, min=0.05, step=0.01), Drop…

help(dx.interactive_linecut_and_colormap)

Help on function interactive_linecut_and_colormap in module dexplore.interactive_linecut_and_colormap:

interactive_linecut_and_colormap(data, initial_var=3, throttle_delay=0.2)
    Generate an interactive colormap from an xarray dataset
    
    Parameters
    ----------
    data:
        A xarray dataset which contains (max) 3 coordinates
    initial_var: (default 3)
        Which dataset variable to start with when opening the widget.
        Either a number index of the dataset variable, or a dataset name. 
    throttle_delay: (default 0.2)
        Throttle time in seconds. To prevent the UI interactions from 
        falling behind, this function throttles calls to the bokeh updates. 
        If your plots are taking a long time to finish updating after 
        moving the slider, you can increase this time.

Contributing

Design philosophy

The idea of writing things like above as functions rather than a integrated UI is that it makes it more modular and easier to customise.

Maybe you have a good idea for another interactive widget? Maybe you can do cool stuff with different libraries? The idea is that you can do this very easily since you just have to write a function that works with the xarray data.

How do I learn this? A good starting point is to have a look at the source files of data explorer. The source files in data explorer are designed to be simple and small: each one contains one (or maybe two) simple functions.

Structure and debugging

Also, as I develop the source files in the Jupyter notebook interface, I also typically debug and test them directly in the source file while I'm working. How does this work? If you open the .py file in the notebook server, then it will look like just a regular notebook. The debugging code is at the bottom in a markdown cell (which gets converted to a special commented region of the code when you save). Simply convert that from a markdown cell to a code cell, and then you can directly in the running kernel debug the function as you develop it. When you're done, simply convert back to a markdown cell and it's ready to be used as a python module.