Getting Started

A quick how-to on the basics of using NDIToolbox

First Run

The first time you start NDIToolbox, it will create a local folder under your $HOME folder (e.g. c:\users\chris in Windows 7/8 or /home/chris under Linux) to store data files, plugins, POD models, and other user-generated files. As of this writing the local folder is by default called nditoolbox (or a7117 in earlier builds, named after the internal TRI/Austin project number), but you can choose any location you like. So after the first run of NDIToolbox you’ll have a local data folder that looks like this:

A configuration file will also be stored under your $HOME folder as nditoolbox.cfg. The configuration file stores basic information about how you're using NDIToolbox, such as how to plot 2D data and where to find your user files. You may also encounter a file nditoolbox.log in the same folder as nditoolbox.cfg, which is a log file used to record errors that occur while using NDIToolbox. The log file is viewable from within NDIToolbox by selecting Display Application Log... from the Help menu. If you run into difficulties with NDIToolbox, you can copy-paste the contents of this log (or add as an attachment of course) to any correspondence with TRI.

Using NDIToolbox

NDIToolbox is designed around the idea of managing data. You have multiple data files organized in a particular way, and you’d like to use NDIToolbox to plot, preview, and manipulate the data. How you organize your data is up to you but one suggestion would be to organize by structure and inspection. For example, you can create a subfolder Pennybacker Bridge under your data folder to house all the inspections of the Pennybacker Bridge; under this subfolder place additional subfolders Bridge Deck and Steel Cables to house inspections of the bridge’s substructures, subfolders under these for particular inspections of the substructure, and so on.

If you don’t want or need this level of organization, you can of course just use c:\users\chris\nditoolbox\data as-is, it’s up to you.

When the user interface is shown you’ll see two panels. The NDE Data panel at the top lists all the files currently in the local data folder.

To refresh the list of data files, press the Refresh Data button in the toolbar; you can also add and remove data from this folder by pressing the + and buttons in the toolbar respectively. Adding a data file prompts you to browse to an existing data file, which will be converted to HDF5 if necessary and copied to the local data folder. Removing a data file removes the file from the local data folder; the original data file isn’t affected.

The Plot panel at the bottom displays thumbnail images of your data as an easy way to browse large sets of data. When you select a data file in the NDE Data panel, NDIToolbox will show the thumbnail plot of this file. The first time you select a file there will be a slight delay in displaying the thumbnail as NDIToolbox is generating it in the background, but after this initial generation it will continue to use this same thumbnail until you Refresh Data or the file’s path changes. If you’d rather not use thumbnails, disable them by toggling the Enable/Disable Thumbnails button.

Working With Data

Importing Data

NDIToolbox uses the Hierarchical Data Format to store data. To import data in other formats (DICOM, DICONDE, delimited text, bitmap images, etc.), select the appropriate menu item from the Import submenu under the File menu, click the Import Data button in the toolbar, or use the keyboard shortcuts CTRL-A or CTRL-+. The file will be read and converted to HDF5 before it is copied to the local data folder. Depending on the type of file being imported, you may be asked to provide additional details - for example, importing an ASCII delimited-text file such as a CSV will bring up a dialog that asks you what character is used as the delimiter, whether lines at the top or bottom should be skipped, whether you'd like to import all or a select number of columns, and so on. If the file contains more than one dataset (for example, an amplitude and a full waveform dataset), NDIToolbox will import each as separate HDF5 files.

The current version of NDIToolbox can import data from the following file types.

Exporting Data

You can also export HDF5 files in the local data folder to a text (CSV) file by selecting the file in the NDE Data panel and selecting the Export Data button (looks like a disk) in the toolbar.

Plots And Previews

Previewing Data

To preview a data file, select it from the NDE Data panel and press the Data button in the toolbar (looks like a table). After loading the data you’ll be presented with a spreadsheet showing the contents.

Working With Three Dimensional Data

NDIToolbox supports previews and plots of three-dimensional data. Throughout this document and in NDIToolbox we refer to X, Y, and Z axes. The following figure illustrates the coordinate system used: Y is the row number, X is the column number, and Z is the third dimension into your data.

For three-dimensional data, you’ll be asked how to take a two-dimensional “slice” for preview. This slice is a plane through the 3D data that’s defined by its orientation and an index. The orientation of the slice specifies which plane your slice will be parallel to: X-Y, X-Z, or Y-Z. The index of the slice specifies how far into the other dimension your slice will be. For example, specifying an orientation of Parallel To X-Y means that your slice will be somewhere in the Z axis, while Parallel to Y-Z will mean your slice will be taken somewhere in the X axis.

If you're familiar with NumPy, here's how the slicing works. For a three-dimensional data set A, a point P at position (x0, y0, z0) is given by P = A[y0, x0, z0] (notice that the Y index is first - NumPy indexes data beginning with row and then column). A planar slice in A is one of the following.

NDIToolbox provides a basic interface for NumPy slicing operations. To slice an existing NDIToolbox dataset, select it in the NDE Data panel and press the Slice Data button in the toolbar (next to the Remove Data button). The Export Slice dialog will be shown, from which you can specify how to slice the data.

For performance reasons by default NDIToolbox doesn't check the size and shape of the data prior to slicing, since this would require reading part of the data and could be a lengthy operation for very large data files. If you don't know the size of the data, you can have NDIToolbox look it up by pressing the Check button. This will also reset the limits of the slice index controls below to the maximum allowable for each dimension of your data, and disable dimensions that are not in the data (e.g., a one-dimensional array will have both X and Z dimensions disabled).

The slice index controls specify how to take the slice in each of the data's dimensions:

In the current version of NDIToolbox arrays are not automatically flattened, which means that the dimensions of the slice are not reduced. For example, if you have a 3D array of size (Y, X, Z) = (350, 250, 100) and take a slice in Z of length 1 you'll have created a new 3D array of size (Y, X, Z) = (350, 250, 1) rather than a 2D array of size (Y, X) = (350, 250).

When you've built your slice, press the Ok button to slice the original data. NDIToolbox will read the slice from the original data and prompt you for a new name for the slice; the slice will be saved as a new HDF5 data file in your NDIToolbox data folder. The slice interface in NDIToolbox is designed to give you a basic user interface for the most common slicing operations in NDE data; for more advanced slicing requirements we recommend a full-blown NumPy session or writing an NDIToolbox plugin.

Plotting Data

To plot a data file, select it from the NDE Data panel and press either the X-Y Plot button (looks like a line chart) or the Image Plot button, depending on whether you want to plot a simple line chart or an image plot. After reading the data NDIToolbox will open a new window with the plot. You can also add/edit plot and axis titles and toggle the grid; image plots additionally allow choosing different color scales and applying labels to the colorbar.

To pan around the plot press the Pan button in the tool palette at the bottom of the plot window, and drag the plot with the left mouse button. You can also zoom in/out in horizontal and vertical scale by dragging with the right mouse button. To zoom in to a particular region, press the Zoom button and select a region in the plot. To reset the plot to its original bounds, press the Home button.

Exporting Plots

If you’d like to export the plot as a static image, press the Save button and choose a destination filename. NDIToolbox currently supports the following formats for exporting plots:

For general distribution a bitmap format such as PNG or JPEG can be used, however if you require higher-quality presentations (e.g., for publication) consider a vector format such as EPS or SVG.

You can also save the data as an HDF5 file by choosing Save Current Data from the File menu.

Plots have various operations such as rectification, window functions, etc., depending on what type of plot is shown. NDIToolbox also supports plugins that can alter the data in a plot. To get back to the original dataset after running any of these operations, choose Revert To Original from the plot window’s Operations menu. This will discard all the changes you’ve made to the data, and replot the original dataset. You can also save a modified dataset by selecting Save Current Data from the plot window’s File menu.

Colormaps

As mentioned above, image plots (and Megaplots, see below) support the use of different color scales ("colormaps") to aid in making the most of the presentation of your data. NDIToolbox's plotting library includes 138 colormaps, but also supports creating your own customized colormaps. To create a new colormap, select the Create Colormap... menu item from the Colormaps submenu in the Plot menu in an image plot or Megaplot presentation. The Create Colormap window shown below will open. Colormaps are created and edited using the controls on the left side of this window; a preview of the current colormap is shown in the image plot on the right. A sample three-color colormap (black, gray, white) is loaded to get you started; you can add/delete/reorder these colors or load your own colormap (more detail below).

A colormap consists of a list of one or more colors defined by their Red, Green, and Blue components and a type. Two types of colormap are supported. A linear gradient colormap linearly interpolates colors between the colors defined in the colormap to create a smooth gradient of 256 colors. A straight list colormap does not interpolate colors and creates a simple list of the colors defined in the colormap.

The following screen shot illustrates the difference between the two types of colormap: on the left, a dataset is shown using the 'cool_blue' colormap that ships with NDIToolbox as a linear gradient. On the right, the same colormap as a simple list. Both colormaps use the same four colors, but the colormap on the left creates gradients between the four defined colors to generate a final colormap with 256 colors. The colormap on the right uses only the four colors defined in the colormap itself.

Most data presentation applications use a linear gradient colormap, however for edge detection and similar "go/no-go" use cases you may find a simple list colormap more useful.

Colors in a colormap are defined as a triplet of 3 numbers that set their Red, Green, and Blue components respectively; and are expressed as a decimal between 0 and 1. The color black is thus defined as 0,0,0; white is 1,1,1; and all other colors are in between these ranges. To add a new color, click the New Item button in the Colors list and the standard color dialog for your operating system will be displayed. Modify the color as desired and click OK; the color's components are added to the colormap. Colors are listed in the colormap from low end of scale to high so that the first color in the list is used to represent the smallest values in the data. To reorder the colors, select a color in the list and press the up or down arrow buttons in the color list to move it up or down in the colormap respectively. To delete a color, select the color and press the delete button in the color list.

To preview the current colormap, press the Preview Colormap button at the top of the Preview pane. The sample data are replotted using your current colormap. Once you're satisfied with your colormap you can save it for future use by selecting the Save Colormap... item from the File menu. By default NDIToolbox will store the colormap in its colormaps folder but you can select any location (note that NDIToolbox will only look in the default folder for colormaps to use, however). Colormaps are saved as ASCII text in JSON and can be edited with a conventional text editor if desired. The filename you choose for the new colormap is used to name the colormap for subsequent plots; as a result you'll likely want to use filenames like "cool blue", "Modified_Spectral," etc. NDIToolbox copies several sample colormaps to your colormaps folder on its first run, we recommend having a look at them to get you started.

Once you've saved your colormap to your colormaps folder, NDIToolbox will now include it in the list of available colormaps. To use your colormap in a plot, select it from the list presented when you select the Select Colormap... menu item from the Colormaps submenu under the Plot menu.

Megaplot

NDIToolbox also has a “MegaPlot” that plots three-dimensional data directly using a common presentation style for NDE data shown below.

In this type of plot, a C-scan plot in the lower-right indicates your position in the Z Axis in the 3D data. To zoom and pan in each plot, make sure the Use Plot Navigation Tools checkbox at the bottom of the screen is checked to enable the tool palette. When this checkbox is unchecked, the tool palette is disabled and the Megaplot presentation responds to clicks inside the plot. For example, when you click on a position (x0, y0) inside the C-scan, the plot will update the other three panels:/p>

By default the Megaplot uses 1D linear slices through the current Cscan for displaying horizontal and vertical B-scans. To use the more conventional 2D planar slices for B-scans, check the Plot Conventional B-scans item from the Plot menu. NDIToolbox will now plot standard 2D B-scans in Megaplot presentations until you uncheck this menu item.

Frequently in ultrasonics NDE you’d like to define a subset of the 3D data to generate a C-scan rather than simply taking a slice in Z. From the Operations menu, choose the Define C Scan menu item. You’ll be asked to specify the range over which you’ll be generating the C-scan, e.g. if you specify 222-444 the C-scan data will be generated using the subset of data between Z=222 and Z=444. Next, you choose the function used to generate the C-scan. Several options are available including min, max, and average. Choose the function and click Ok, and the C-scan data will now be the results of applying this function against the specified subset of data. For example, the following screenshot shows the results of applying the min function against Z=222:444 in the ultrasonic immersion tank data shown above in the MegaPlot screenshot.

Notice that when you generate a C-scan plot in this fashion that the linear B-scan plots are now taking data from this C-scan and not the original data. The A-scan continues to take data from the original data set, as will conventional B-scans.

One thing to note about MegaPlots is that although your data is presented in one or two dimensions, the underlying 3D data is unchanged. Plugins that don't expect a 3D data set can behave strangely or run slowly. If you need to use a plugin and you're not sure if it supports 3D data, consider plotting a slice of the 3D data instead of using the MegaPlot and running your plugin against this plot.

Although the MegaPlot offers one way to present 3D data by showing A, B, and C scans simultaneously, you can generate each of these plots by itself in its own window. To produce a B-scan plot from a 3D dataset for example, choose the Image Plot function in NDIToolbox and as usual with 3D data, specify a plane. Going back to the immersion tank ultrasonic data shown in the MegaPlot figure, the following screenshot shows two B-scans taken from this data by slicing in the X and Y axes.

Similarly, if you'd like to generate an A-scan of a 3D data set, choose the X-Y Plot function. You'll be asked to specify a slice similar to the slicing done for image plots, but in this case you specify two indices. A typical A-scan representation in NDIToolbox is performed by specifying a point in X and Y and plotting the Z values at that point, as shown in the following A-scan plot taken from the immersion tank scan data at position P(267, 152).

Batch Mode

If you have multiple data files to analyze it can prove time-consuming to import each dataset, plot, execute your analysis, and save the output. NDIToolbox "batch mode" can streamline this procedure by allowing you to specify a plugin to run and a set of files to analyze without user interaction. To use batch mode, open your operating system's console in the NDIToolbox program folder, and run python nditoolbox.py -h to print the basic usage instructions. You'll see something like the following.

      NDIToolbox Batch Mode
      usage: nditoolbox.py [-h] [-t TOOLKIT] [-c TOOLKIT_CONFIG] [-f FILETYPE]
                           [-i ...] [-s]

      Run NDIToolbox toolkit in batch mode

      optional arguments:
        -h, --help            show this help message and exit
        -t TOOLKIT, --toolkit TOOLKIT
                              Name of toolkit to run
        -c TOOLKIT_CONFIG, --toolkit_config TOOLKIT_CONFIG
                              Config file for toolkit
        -f FILETYPE, --filetype FILETYPE
                              Specify type of input file (default: guess from file
                              extension)
        -i ..., --input_files ...
                              Specify input file or files
        -s, --save_output     Save plugin output to new HDF5 data file
        -m, --multiprocess    Use multiple simultaneous processes for analysis
    

At a minimum, you'll need to specify the toolkit to run and the file(s) to run it on. Use -t TOOLKIT to specify the toolkit name - note that you specify the actual name of the toolkit and not its filename, e.g. -t MedianFilterPlugin and not -t medfilterplugin.py. To specify the file(s) to analyze, use -i followed by a wildcard pattern, e.g. -i data/ut_tests/*.csc to run the plugin against all files found in the data/ut_tests folder with the .csc extension (commonly used for UTWin data files). For each file that matches your list of specified files, the data will be read, fed to your chosen toolkit, and (optionally) saving the resultant data to a new HDF5 data file in your local data folder.

When specifying input files, if you use automatic file format detection (more below), you don't have to run batch mode separately for every file format you have. Just specify the folder with the data - for each file in the folder, if its format is supported by NDIToolbox and recognized, it will be read successfully.

Normally, NDIToolbox attempts to load data files based on their file extension rather than asking you about the file format. If you need to override this feature, you can specify the file import format to use with the -f switch. As of this writing, the following file format specifiers are supported.

It's important to remember that in using the -f switch, you are overriding NDIToolbox's attempts to choose the file format automatically. If you use it, be sure that all your input files are of the format you specify! Any files that don't match the format you specify will not be read successfully. Unless you really need it, we recommend letting NDIToolbox make the determination on file format.

If your toolkit requires configuration, you can specify a configuration file to use with the -c switch. This file should be a JSON file that lists the configuration required for the toolkit. Finally, to save the results of each analysis, add the -s switch. Unless your toolkit saves its own results you'll likely want to specify this switch as otherwise no data are saved!

For some types of input file (UTWin and Winspect as of this writing), it's not uncommon to have several datasets in each file. If multiple datasets are found in a file, NDIToolbox will attempt to read all of them and pass them to the toolkit. Output files will be named using the base name of the current input file with an appended string indicating the dataset, e.g. a file scan1.sdt containing one Time Of Flight, one amplitude, and two waveform datasets would produce output files named scan1_tof0.hdf5, scan1_amplitude0.hdf5, scan1_waveform0.hdf5, and scan1_waveform1.hdf5. However, your toolkit needs to have been written to support this feature - contact the toolkit's developer for details. The MedianFilterPlugin and the NormalizePlugin that ship with NDIToolbox both support multiple datasets.

Multiprocess Mode

Batch mode also supports symmetric multiprocessing (SMP) of input files with the -m switch, in which two or more concurrent processes conduct your analysis. In normal single-process mode, each input file is analyzed in sequence; in multiprocessing mode, NDIToolbox creates one worker process for each "core" (or CPU) in your computer and each worker conducts analysis of an input file. Since two or more input files can now be analyzed simultaneously, this can dramatically improve analysis speeds. In one test using a quad-core desktop machine and 39 input files, TRI measured an average analysis time of 4:45 (7 seconds per input file) in single-process mode vs. an average time of 1:26 (2 seconds per input file) in multiprocess mode. Although this is just one result and your times may differ, it does illustrate the benefits of multiprocess mode for some scenarios.

If you use multiprocess mode, be aware that memory requirements are now much higher since you're essentially trying to keep multiple datasets in memory at once. For this reason you should avoid multiprocess mode for large inputs, which essentially means inputs whose size is on the order of your machine's available RAM (not your total RAM) divided by the number of cores in your machine. For example, on an 4-core machine with 8GB of RAM, your operating system may take 2.5GB or more of RAM, leaving you with 5.5GB of available memory. In this case, you should avoid multiprocess mode if your input files are around 1.3GB or so in size, since this will put you in danger of exhausting available memory.

That said, multiprocess mode is worth investigating if you have multiple smallish input files to process. It can offer significant performance improvements under the right conditions. If you encounter memory issues or don't see any performance improvement, simply omit the -m switch for future batch mode runs.

Example

python nditoolbox.py -t MedianFilterPlugin -c ~/tmp/bane_batch_tests/medfiltercfg.json -s -i ~/tmp/bane_batch_tests/*.csc

Runs the MedianFilterPlugin on all files in the ~/tmp/bane_batch_tests/ folder with the .csc file extension (note that this is case sensitive). The median filter is using the JSON file ~/tmp/bane_batch_tests/medfiltercfg.json as its configuration file; the contents of which specify a rank 5 filtration:

    # medfiltercfg.json
    # JSON configuration for Median Filter NDIToolbox toolkit
    # Accepted parameters:
    #     "kernel size"  :  rank of median filter (defaults to 3)
    {"kernel size":5}
    

As no file format was specified, NDIToolbox will assume the files are UTWin CScan files based on their file extension of .csc; output of the median filtration will be saved to the batch_output folder in your local data folder.

Next Steps

The best way to learn your way around NDIToolbox is to experiment. Try importing your data and test out an organization scheme. Plot, preview, slice. When you're familiar with the basics, the next step will be to write your own plugins...