In the development of experimental infrastructure, software is an important element which is often overlooked. For X-ray Coherent Diffractive Imaging (CDI), in particular, computational inversion of the measured diffraction intensity is an integral part of the image reconstruction process. The proliferation of phase retrieval algorithms has created the need for common software tools which encompass all these approaches. The NADIA (Nadia's Algorithms for Diffractive Imaging Analysis) software package aims to provide a set of phase retrieval tools which are robust, efficient and available to the wider CDI community.
In this documentation, we introduce the software package, NADIA, which is able to reconstruct and simulate X-ray diffraction data for plane-wave, Fresnel, ptychographic, phase-diverse and partially coherent (both spatially and temporally) CDI. To the best of our knowledge, this is the first standard software package which includes all of these different approaches. The software allows for any choice of iterative algorithm, including customised algorithms, and also includes the 'shrinkwrap' algorithm. For Fresnel CDI, additional complex constraints on the transmission function may also be used. The software consists of a C++ library, binding to IDL and command-line tools, in addition to example code and documentation. It has been tested on the Mac, Windows and Ubuntu operating systems. It is issued under the GNU public license.
It is assumed that readers of this manual are familiar with the relevant literature describing the CDI algorithms implemented in and we will therefore not repeat this information here. Please refer to any of the reviews for more information on CDI. We intend that this manual is used in the following ways:
The software is available for download from the software website as either,
For Windows users who only wish to use the IDL capabilities or command-line tools we recommend downloading the Windows binary file. For all other users, we recommend downloading the source code package.
Currently the code depends on several other libraries which will need to be installed prior to installing this package:
Ubuntu users can easily install the external packages using the synaptic package manager . The packages are named libfftw3-dev, libtiff4-dev, liblapack, and libhdf4-dev.
Next, download our source code from the website and put it into an appropriate directory. Unzip and untar the source code:
gunzip NADIA.0.5.tar.gz tar -xvvf NADIA.0.5.tar
Change into the directory NADIA and compile the source code:
cd NADIA ./configure make
If ./configure doesn't finish by making a set of "Makefiles", the dependent libraries probably haven't been installed correctly or ./configure does not know their location.
The result of compilation (if successful) are libraries and header files (in the /lib and /include directories) and some example programs which are ready to execute.
The source files in the examples/ directory e.g. PlanarCDI_example.c demonstrate how the libraries can be used within a user's own code. The Makefile in examples/ shows how a user's code can be compiled and the libraries linked.
./configure --with-fftw_inc=/sw/include --with-fftw_lib=/sw/lib CXX=''g++ -arch i386'' --with-mfhdf_inc=/sw/include --enable-checkingThe ``-with'' flags will tell the compiler where to look for the external packages. The CXX="g++ -arch i386" relates to your computer's architecture. This is for a late 2010 Macbook pro running 10.6, try this, if it doesn't work, try looking up the relevant information for your computer.
Cygwin provides a Linux/Unix type environment for Windows (the interface is basically a Unix style terminal). Installing the NADIA source code in this environment is very similar to installing it in Linux/Unix and (in my opinion) makes modifying and recompiling the code a bit easier than it would be in Windows otherwise. First you will need to install Cygwin. You can do this by going to http://www.cygwin.com/, scrolling to the bottom of the page and clicking on "Install or update now (using setup.exe)". Open setup.exe and follow the installation instructions. Apart from the "Package Selection" screen, used the default options. When you get to the "Package Selection" screen leave "All" on "Default" and add the additional packages by searching for them and then clicking on them:
Note that you can update the packages which are installed after you've installed Cygwin just by clicking on setup.exe again (it remembers which are installed). If you get an error during the post-installation phase (error code 128) it maybe due to a virus checker interfering with the Cygwin installation.
Install the HDF4 libraries. There is no package for this in the package manager so go to http://www.hdfgroup.org/release4/obtain.html and download their Windows binary for your system (32 or 64 bit). Unzip the file using the standard Windows tool and install to a directory without any spaces (Cygwin sometimes struggles with spaces)
Some of the HDF files are named in such a way that the NADIA cannot find them. So go in to the bin directory of your HDF install (something like c:\HDF_Group\HDF\4.2.8\bin) and create some dynamic links:
ln -sf mfhdfdll.dll libmfhdf.dll ln -sf hdfdll.dll libdf.dll ln -sf szip.dll libsz.dll
Next, download NADIA here and put it into your home
directory on Cygwin
(C:\cygwin\home\your_user_name). Open Cygwin and unzip and untar the package:
$ gunzip NADIA.0.5.tar.gz $ tar -xvf NADIA.0.5.tar
When you type ``ls'' you should see the directory "NADIA" plus a few others. Change into the NADIA directory:
$ cd NADIATell the package where your hdf and szip libraries files are located (replace the "user_name" with your cygwin user name):
$ ./configure --with-mfhdf_lib=c:/HDF_Group/HDF/4.2.8/bin --with-mfhdf_inc=c:/HDF_Group/HDF/4.2.8/include --with-sz_lib=c:/HDF_Group/HDF/4.2.8/bin --with-df_lib=c:/HDF_Group/HDF/4.2.8/bin
If you get an error, check that the paths given to configure are correct; the lib directories should have files starting with "lib" and the include directory should have files ending in ".h". If configure worked correctly, compile the libraries and examples:
Hopefully you won't get any errors and when you look in the lib/ and include/ directories you will see some files. The libraries can be useful if you ever need to create your own code (similar to the examples). The examples show how the libraries can be used.
and run one of the examples. e.g.:
If everything is working correcting, you should see some text output (counting the iterations). If you look at this directory in the Windows browser you should see that some .ppm image files are created which give you the reconstruction image for various number of iterations.
Have a look at the code in more detail to see what's going on:
$ xemacs PlanarCDI_example.cand edit some variables to see what happens (eg. change the reconstruction algorithm). Then recompile and rerun the example:
$ make $ ./PlanarCDI_example.exe-------------
Download the package (binary file) from the main webpage. Executing the .msi file should automatically start the installation. Select which directory you want the files to be installed in (if it is not your own machine you probably won't have permission to install into ``Program Files''). Once installed you can start using the library and command line executable files. Look at the README file which comes with the distribution to find out more.
To uninstall, go to "Control Panel", go to "Add/Remove program", find NADIA and remove it.
Note: this section is only applicable to users who are members of the ARC Centre of Excellence in Coherent X-ray Science (CXS).
If you want to quickly use the software without the hassle of installing many packages, the simplest way is by logging into Osiris and using it there. This will also be useful to know how to do in the long run because the computing power available there is more than your laptop/desktop. The first step is to ask for an account on OSIRIS if you don't already have one (and are a member of CXS). Then follow the steps below for logging in, and for setting up and running the software.
Open a terminal and type:
ssh -X11 -2 osiris.ph.unimelb.edu.auThe "-X11 -2" enables you to view graphics on osiris.
On Windows, you will need to install an ssh client. Many people (including me) recommend the program putty for this. After you install it, test it works by logging into osiris:
You are now half way there. You can log in, but you won't be able to view any images which are stored on osiris yet. To be able to do this, you need to install an X-windows server (graphical information can be passed back to your computer once you have this). I found Xming to be good. I installed it with the default options ("normal Putty").
I also found that I need to install Xming-fonts in order for the "emacs" text editor to display fonts correctly. During the set-up I ticked all the boxes so that all the fonts were installed.
Now, each time you want to log into osiris you will need to run Xming first. It will run in the background, so all you'll see when it's running is a small icon on the tool bar. You can close it once you've finished with your putty session.
Finally go into putty again, and load your "osiris" session.
You now have available all of the command line tools listed here. Note that this will only work for your current session on osiris. To make it permanent, open the file ".bash_profile" in your home area:
emacs .bash_profileAdd the line above to the end of the file, save and then close it (use the menu or type "xs" followed by "xc" while holding down the Ctrl key).
If you plan to work with IDL you will also need to update your LD_LIBRARY_PATH with the wrapper library path:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/DIXE/dixe/tags/rel_0.5/NADIA/interfaces/idl/and update the IDL_PATH with the IDL module location:
The simplest way to run reconstruction is to use the CDI_reconstruction.exe command line tool. It takes three arguements as input:
Test out the program by running it with one of the example configuration files:
./CDI_reconstruction.exe planar_example.configYou should see output telling you the iteration number and the error. The error is a measure of the difference between the diffraction data and the reconstructed diffraction (the usual chi squared given in the literature). You can kill the process at any time by typing Ctrl-C or you can wait for it to finish if you prefer.
Now look into the configuration file by typing:
emacs planar_example.configYou can see that the data and support file names are given. Type:
gthumb image_files/planar_support.tiffto view the support. The data file can't be viewed with gthumb because it's in binary format, however you can convert it to a .ppm file using one of the other command line tools and then view it:
dbin2ppm.exe image_files/planar_data.dbin planar_data.ppm 1024 1024 gthumb planar_data.ppm
The CDI_reconstruction.exe program outputs a .ppm file of the current magnitude of the estimated exit surface wave every 40 iterations. Type ``ls'' again to see a list of these files (they should look something like planar_x.ppm, where x is the iteration number). View them using gthumb. The reconstruction is slowly converging. Note that it may not completely converge as the number of iterations performed in this example is a lot less than usually used. The exit surface wave of the final iteration is saved as a complex binary file (with the file extension .cplx). You can view the magnitude and phase of the wave, by using the cplx2ppm.exe command, followed by the gthumb command. You can also use this file as the starting point for another reconstruction (see planar_example.config for instructions).
Now, try editing the configuration file:
emacs planar_example.confige.g. with a different combination of reconstuction algorithms, a different output rate, support file or shrink-wrap parameters etc. and see what happends.
A similar example exists for the Fresnel case. This can be run by first reconstructing the white field using 3-plane propogation:
./CDI_reconstruction.exe fresnel_example.config fresnel_wfThen reconstruct the sample:
./CDI_reconstruction.exe fresnel_example.config fresnel
The command line tools are useful to get started quickly, but if you want full flexibility with the code, it's best to learn a little bit of "c" and examples are provided for this as well. The same reconstruction as planar.config performs can be done using PlanarCDI_example.c. Compile this code using the Makefile:
makeNow when you type ls you should see a filenamed PlanarCDI_example.exe. Run this.
./PlanarCDI_example.exeYou should find it gives almost exactly the same output as the CDI_reconstuction program did. Have a look at the source code: emacs PlanarCDI_example.c edit it and recompile using make to see what happens. Similarly, have a look at and Fresnel example: FresnelCDI_WF_example.c and FresnelCDI_example.c, and a simulation of the object from the planar example: PlanarCDI_simulation_example.c.
Later, if you write your own "c" code, it's easy to compile and link it with the libraries using the "Makefile" from the examples directory. Edit the "EXAMPLE_SRC" variable in the Makefile with the name of your ".c" file, then type "make" on the command line to compile it.
Some IDL wrappers have been written for the C++ library functions. This is useful for people who are more familiar with IDL than C, or want to take advantage of the graphing abilities of IDL. It will give greater flexibility than the command line program but doesn't allow you to use all the features of the C++ library.
Begin by copying the IDL examples to your working directory,
cp /data/DIXE/dixe/tags/rel_0.5/NADIA/interfaces/idl/*_example.pro ./and run them:
Look at the .pro files to see what they do.
Open a terminal and follow the instruction given for ``Users on Osiris'', from Section 3.1.3 onwards. Replace the path to the software on osiris, ``/data/DIXE/dixe/tags/rel_0.5/NADIA/'' in the instructions, with the path where you installed the software.
The software doesn't have a flash graphical user interface, but rather is a library and set of files which can be used to performing image reconstruction through various means: IDL, C++ and command lines tools. Brief instructions for how to get started for each of these options is described below.
.Compile NADIA_interface.proin IDL. Documentation for each of the routines can be found in Appendix B and on the website and examples are provided in the examples subdirectory of this distribution.
To run the IDL examples:
C:\<path you installed the package>\NADIA\examples\image_files\<file_name>Note how the "/"s should be changed to `''\''s.
Some command line tools are provided for image reconstruction and for converting between different image file formats. The image reconstruction tool (CDI_reconstruction.exe) is not very flexible (for example it doesn't allow simulation to be carried out). So in general the IDL option is recommended. However, it does not require any extra software to be installed, for example IDL or mingw, so maybe useful, to quickly demonstrate the concepts of the algorithms to new users.
A description of each of the command line tools can be found in Appendix C.
If you wish to run the software using C++ you should either be familiar with C/C++ or be willing to invest a bit of time to understand how to compile and write this type of code. For this reason it's not advised that a beginner attempts to run the C++ examples under Windows (unless in Cygwin). However, the benefit of using C/C++ is that it is more powerful and you will have greater flexibility with the software library. If your reconstruction requires a lot of computing power (for example you plan on run on osiris or similar, rather than a desktop), this is the best option in our opinion.
Five examples showing how to write c code using the NADIA software for CDI reconstruction and simulation are given in the ``examples'' directory. The header files for the NADIA software project can be found in the "include" directory and the libraries in the "bin" (libraries which the NADIA software depends on are also provided in this directory.
We will leave the details of how to compile in Windows to the users, as this will be compiler and system dependant. However, we do recommend compiling under mingw with msys because it is free; the code provided here was compiled with it, so I know it works; and because compilation is similar to linux/unix meaning you will be better trained for compiling on osiris or similar if the need arises.
The package consits of a set of examples which demonstrate how to use the library and tools of the software package for various reconstructions and simulations. Most are located in the examples subdirectory of the package.
The examples should be compiled automatically after installation (Section 2). To recompile (e.g. after making modifications to the examples), you may type "make" in the examples directory. To force a recompilation type "make clobber" followed by "make".
In this section, we describe how a number of common tasks can be achieved using the software. More detail of the functionality available can be found in the doxygen documentation () and IDL routine documentation ().
[language=IDL] data = NADIA_READ_PPM(1024, 1024, 'data_file.ppm')etc. or use one of IDLs built in libraries to get the input in matrix form.
The reconstruction code must be initialised for either
plane-wave, Fresnel white-field, Fresnel or phase diverse
[language=C++] //Create a complex 2D field which will //hold the result of the reconstruction. Complex_2D object_estimate(nx,ny); //Create the planar CDI object which will //be used to perform the reconstruction. PlanarCDI planar(object_estimate); //set the support and intensity planar.set_support(support,false); planar.set_intensity(data);In IDL
[language=IDL] NADIA_INIT_PLANAR, data, support
The software is capable of accepting data in hdf, tiff or ppm image format. It can output as a grey scale tiff or ppm image with X bytes per pixel. In addition data can be read and stored as a binary file of doubles. This may be useful in situations where high prescision is required. Similarly, two dimensional arrays of complex numbers, such as those produced in the reconstruction, may be stored as a binary file of fftw complex number types.
[language=C++] //read in the data Double_2D data; read_image(``data_file_name.tiff'', data); //read in the support shape Double_2D support; read_image(``support_file_name.tiff'', support);
[language=IDL] data = NADIA_READ_PPM(1024, 1024, 'data_file.ppm')etc. or use one of IDLs built in libraries to get the input in matrix form.
[language=C++] //write out the result: // - as a complex binary file write_cplx(temp_str.str(),object_estimate); // - just the magnitude as an image Double_2D object_mag; object_estimate.get_2d(MAG,object_mag); write_image("mag.tiff",object_mag);
[language=IDL] ; Record the complex field to a binary file NADIA_WRITE_CPLX, result, 'result_file.cplx'The magnitude and phase of the complex matrix can be viewed and saved using the standard IDL routines.
Whether it is performed in C++ or IDL, a plane-wave, Frensel and Fresnel white-field reconstruction consists of the following basic steps:
[language=IDL] ; Perform one iteration. The exit- ; surface-wave is return to 'result' result = NADIA_ITERATE() ; Perform 50 iterations result = NADIA_ITERATE(50)
Various aspects of the reconstruction can be controlled, for example, the algorithm, the relaxation parameter and the shrinkwrap algorithm. How this can be achieve with the software is described in this section.
By default plane-wave CDI uses the ... algorithm and Fresnel CDI uses the ... algorithm. This can be easily changed at any point in the reconstruction using the method
void PlanarCDI::set_algorithm(int alg)Where the algorithm options are:
For example, if you have an object of type FresnelCDI called ``fcdi'', and you wish to change the algorithm to HIO:
In addition to this extensive list, custom algorithms can be used. Iterative reconstruction algorithms can be expressed as a combination of the following 5 operators: , , , and Where is the support projection and is the intensity projection. These can be combined to form a basis of 10 vectors (see Harray's review). The parameters to this method set the coefficient for each of these vectors. Note: the vector set is linearly dependant.
Please see page 22 of the H.M. Quiney review: TUTORIAL REVIEW, Coherent diffractive imaging using short wavelength light sources, Journal of Modern Optics, 2010, DOI: 10.1080/09500340.2010.495459
void Planar::set_custom_algorithm (double m1, double m2, double m3, double m4, double m5, double m6, double m7, double m8, double m9, double m10 )
Parameters: m1 Coefficient to the term m2 Coefficient to the term m3 Coefficient to the term m4 Coefficient to the term m5 Coefficient to the term m6 Coefficient to the term m7 Coefficient to the term m8 Coefficient to the term m9 Coefficient to the term m10 Coefficient to the term
In IDL, the routine ... can be called to select the algorithm.
In designing the code we have tried to find a good balance between optimising the speed, reducing the memory used, and keeping the code modulare and simple to read. There is always room to improve the performance and we provide the figures below as a benchmark of the current state of the software in version . These values are independante of whether C/C++, IDL or the command-line tools are used.
By default the fast fourier transform library used with is single precision. We found no noticible lose in performance or correctness by using single precision, however users may choose to use double precision fast fourier transforms by adding the flag ``-enable-double-precision'' to ``./configure'' during installation. The class Double_2D, despite its name, will also use the same precision as the fourier transform library and is single precision by default.
For the sake of advanced users we provide a description of the software library structure in this section. Figure 1 shows the relationship between the non-I/O class of the software, and lists the public member functions.
The reconstruction classes PlanarCDI, FresnelCDI and FresnelCDI_WF (shows in the shaded region of Figure 1, all inherit from the abstract base class, BaseCDI. Common functionality, for example the definition of the reconstruction algorithms is provided in BaseCDI. The reconstruction type PhaseDiverseCDI, which is capable of performing phase-diverse and ptycographic reconstruction accepts multiple objects of type BaseCDI.
my_planar_recon.set_support(support) my_planar_recon.set_intensity(data) my_planar_recon.initialise_estimate()
my_planar_recon.set_algorithm(HIO) my_planar_recon.set_relaxation_parameter(0.95)The algorithm is fully customisable through the BaseCDI::set_custom_algorithm method. See page 22 of the H.M. Quiney review: TUTORIAL REVIEW, Coherent diffractive imaging using short wavelength light sources, Journal of Modern Optics, 2010, DOI: 10.1080/09500340.2010.495459 for the formalism.
my_planar_recon.iterate()This will automatically update the values in the Complex_2D estimate and the current error. The error can be retrieved using BaseCDI::get_current_error().
For cases where the beam-stop region must be excluded during reconstruction, you should call the set_beam_stop() method, to define the beam-stop region before running iterate().
This class can be used to perform phase diverse or ptychographic reconstruction of either Fresnel or plane-wave CDI data. Any number of frames (also called local/single frames or probes in this documentation) may be added to the reconstruction. In order to perform a reconstruction, you will need to create either a new FresnelCDI or PlanarCDI object for each of these 'local' datasets. The FresnelCDI or PlanarCDI objects must then be passed to a PhaseDiverseCDI object. Because the each sub-iteration (ie. each iteration of a local frame) is performed using the code in FresnelCDI/PlanarCDI, all the functionality available in these classes is also available here. For example complex constraint can be set, shrink-wrap can be used, the estimate can be initialised using the results from a previous reconstruction etc. An example can be found in the /examples directory, demonstrating how to use PhaseDiverseCDI.
The displacement in position between 'local' frames may be either transverse to the beam direction, or longitudinal, along the beam direction. The longitudinal position (for FresnelCDI) is set when the FresnelCDI objects are initially constructed. Transverse positions are set when the FresnelCDI/PlanarCDI objects are added to the PhaseDiverseCDI object. This class allows the transverse positions to be automatically adjusted during reconstruction using the "adjust_positions" function.
The code allows for several options in the type of reconstruction which is done:
where 5#5 is the updated global function, 6#6 is the updated local function for the nth local frame. 7#7 is the relaxation parameter and the weight is 8#8 for Fresnel CDI or 9#9 for Plane-wave CDI. The weight is zero outside of the support.
where 5#5 , 6#6 , and 7#7 were defined earlier. The weight, w, is similar to that used for series reconstruction, but the weight it normalised such that 11#11 . i.e. 12#12
This class is based on the paper "Use of a complex constraint in coherent diffractive imaging", J. N. Clark et. al., 2010. The notation used there is also used here. Users should have knowledge of this or similar papers before using this class.
A section of the reconstructed transmission function will be updated according to a restriction on the value of c (c = beta/delta). If the material is of a known element, then c can be fixed to the know value, and the phase and magnitude of the transmission function updated accordingly. Alternatively, c can be left to float, and calculated for each iteration from a mean over the defined region. The parameters alpha1 and alpha2 are used to control the strength of the constraint.
Instances of this class should be pass to a TransmissionConstraint object. If a sample consists of different regions which are locally homogeneous in element, multiple instances of this class should be created, each one defining a different region of the sample, and then each should be added to the TransmissionConstraint object.
Copyright (C) 2011 The ARC Centre of Excellence in Coherent X-ray Science.
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program (see COPYING). If not, see <http://www.gnu.org/licenses/>.
We also ask that you cite this software in publications where you made use of it for any part of the data analysis.