Asynchronous File I/O

Synchronous I/O means that the method is blocked until the I/O operation is complete, and then the method returns its data. With asynchronous I/O, a user can call BeginRead. The main thread can continue doing other work, and later the user will be able to process the data. Also, multiple I/O requests can be pending simultaneously.

To be informed when this data is available, you can call EndRead or EndWrite passing in the IAsyncResult corresponding to the I/O request you issued. You can also provide a callback method that should call EndRead or EndWrite to figure out how many bytes were read or written. Asynchronous I/O can offer better performance when many I/O requests are pending simultaneously, but generally requires some significant restructuring of your application to work correctly.
The Stream class supports the mixing of synchronous and asynchronous reads and writes on the same stream, regardless of whether the operating system allows this. Stream provides default implementations of asynchronous read and write operations in terms of their synchronous implementations, and provides default implementations of synchronous read and write operations in terms of their asynchronous implementations.
When implementing a class derived from Stream, it is necessary to provide an implementation for either the synchronous or the asynchronous Read and Write methods. While overriding Read and Write is permissible, and the default implementations of the asynchronous methods (BeginRead, EndRead, BeginWrite, and EndWrite) will work with your implementation of the synchronous methods, this does not provide the most efficient performance. Similarly, the synchronous Read and Write methods will work correctly if you provide an implementation of the asynchronous methods, but performance is generally better if you specifically implement the synchronous methods. The default implementations of ReadByte and WriteByte call the synchronous Read and Write methods with a one-element byte array. When deriving classes from Stream, if you have an internal byte buffer, it is strongly recommended that you override these methods to access your internal buffer for better performance.
A stream that connects to a backing store overrides either the synchronous or asynchronous Read and Write methods to get the functionality of the other by default. If a stream does not support asynchronous or synchronous operations, the implementer need only make the appropriate methods throw exceptions.
The following example is an asynchronous implementation of a hypothetical bulk image processor, followed by a synchronous implementation example. This code is designed to perform a CPU-intensive operation on every file in a directory. For more information, see the Asynchronous Programming Design Patterns topic.

using System;
using System.IO;
using System.Threading;
using System.Runtime.InteropServices;
using System.Runtime.Remoting.Messaging;
using System.Security.Permissions;
using Microsoft.Win32.SafeHandles;

public class BulkImageProcAsync
{
    public const String ImageBaseName = "tmpImage-";
    public const int numImages = 200;
    public const int numPixels = 512 * 512;

    // ProcessImage has a simple O(N) loop, and you can vary the number
    // of times you repeat that loop to make the application more CPU-
    // bound or more IO-bound.
    public static int processImageRepeats = 20;

    // Threads must decrement NumImagesToFinish, and protect
    // their access to it through a mutex.
    public static int NumImagesToFinish = numImages;
    public static Object[] NumImagesMutex = new Object[0];
    // WaitObject is signalled when all image processing is done.
    public static Object[] WaitObject = new Object[0];
    public class ImageStateObject
    {
        public byte[] pixels;
        public int imageNum;
        public FileStream fs;
    }

    [SecurityPermissionAttribute(SecurityAction.Demand, Flags=SecurityPermissionFlag.UnmanagedCode)]
    public static void MakeImageFiles()
    {
        int sides = (int)Math.Sqrt(numPixels);
        Console.Write("Making {0} {1}x{1} images... ", numImages,
            sides);
        byte[] pixels = new byte[numPixels];
        int i;
        for (i = 0; i < numPixels; i++)
            pixels[i] = (byte)i;
        FileStream fs;
        for (i = 0; i < numImages; i++)
        {
            fs = new FileStream(ImageBaseName + i + ".tmp",
                FileMode.Create, FileAccess.Write, FileShare.None,
                8192, false);
            fs.Write(pixels, 0, pixels.Length);
            FlushFileBuffers(fs.SafeFileHandle);
            fs.Close();
        }
        fs = null;
        Console.WriteLine("Done.");
    }

    public static void ReadInImageCallback(IAsyncResult asyncResult)
    {
        ImageStateObject state = (ImageStateObject)asyncResult.AsyncState;
        Stream stream = state.fs;
        int bytesRead = stream.EndRead(asyncResult);
        if (bytesRead != numPixels)
            throw new Exception(String.Format
                ("In ReadInImageCallback, got the wrong number of " +
                "bytes from the image: {0}.", bytesRead));
        ProcessImage(state.pixels, state.imageNum);
        stream.Close();

        // Now write out the image.  
        // Using asynchronous I/O here appears not to be best practice.
        // It ends up swamping the threadpool, because the threadpool
        // threads are blocked on I/O requests that were just queued to
        // the threadpool. 
        FileStream fs = new FileStream(ImageBaseName + state.imageNum +
            ".done", FileMode.Create, FileAccess.Write, FileShare.None,
            4096, false);
        fs.Write(state.pixels, 0, numPixels);
        fs.Close();

        // This application model uses too much memory.
        // Releasing memory as soon as possible is a good idea, 
        // especially global state.
        state.pixels = null;
        fs = null;
        // Record that an image is finished now.
        lock (NumImagesMutex)
        {
            NumImagesToFinish--;
            if (NumImagesToFinish == 0)
            {
                Monitor.Enter(WaitObject);
                Monitor.Pulse(WaitObject);
                Monitor.Exit(WaitObject);
            }
        }
    }

    public static void ProcessImage(byte[] pixels, int imageNum)
    {
        Console.WriteLine("ProcessImage {0}", imageNum);
        int y;
        // Perform some CPU-intensive operation on the image.
        for (int x = 0; x < processImageRepeats; x += 1)
            for (y = 0; y < numPixels; y += 1)
                pixels[y] += 1;
        Console.WriteLine("ProcessImage {0} done.", imageNum);
    }

    public static void ProcessImagesInBulk()
    {
        Console.WriteLine("Processing images...  ");
        long t0 = Environment.TickCount;
        NumImagesToFinish = numImages;
        AsyncCallback readImageCallback = new
            AsyncCallback(ReadInImageCallback);
        for (int i = 0; i < numImages; i++)
        {
            ImageStateObject state = new ImageStateObject();
            state.pixels = new byte[numPixels];
            state.imageNum = i;
            // Very large items are read only once, so you can make the 
            // buffer on the FileStream very small to save memory.
            FileStream fs = new FileStream(ImageBaseName + i + ".tmp",
                FileMode.Open, FileAccess.Read, FileShare.Read, 1, true);
            state.fs = fs;
            fs.BeginRead(state.pixels, 0, numPixels, readImageCallback,
                state);
        }

        // Determine whether all images are done being processed.  
        // If not, block until all are finished.
        bool mustBlock = false;
        lock (NumImagesMutex)
        {
            if (NumImagesToFinish > 0)
                mustBlock = true;
        }
        if (mustBlock)
        {
            Console.WriteLine("All worker threads are queued. " +
                " Blocking until they complete. numLeft: {0}",
                NumImagesToFinish);
            Monitor.Enter(WaitObject);
            Monitor.Wait(WaitObject);
            Monitor.Exit(WaitObject);
        }
        long t1 = Environment.TickCount;
        Console.WriteLine("Total time processing images: {0}ms",
            (t1 - t0));
    }

    public static void Cleanup()
    {
        for (int i = 0; i < numImages; i++)
        {
            File.Delete(ImageBaseName + i + ".tmp");
            File.Delete(ImageBaseName + i + ".done");
        }
    }

    public static void TryToClearDiskCache()
    {
        // Try to force all pending writes to disk, and clear the
        // disk cache of any data.
        byte[] bytes = new byte[100 * (1 << 20)];
        for (int i = 0; i < bytes.Length; i++)
            bytes[i] = 0;
        bytes = null;
        GC.Collect();
        Thread.Sleep(2000);
    }

    public static void Main(String[] args)
    {
        Console.WriteLine("Bulk image processing sample application," +
            " using asynchronous IO");
        Console.WriteLine("Simulates applying a simple " +
            "transformation to {0} \"images\"", numImages);
        Console.WriteLine("(Async FileStream & Threadpool benchmark)");
        Console.WriteLine("Warning - this test requires {0} " +
            "bytes of temporary space", (numPixels * numImages * 2));

        if (args.Length == 1)
        {
            processImageRepeats = Int32.Parse(args[0]);
            Console.WriteLine("ProcessImage inner loop - {0}.",
                processImageRepeats);
        }
        MakeImageFiles();
        TryToClearDiskCache();
        ProcessImagesInBulk();
        Cleanup();
    }
    [DllImport("KERNEL32", SetLastError = true)]
    private static extern void FlushFileBuffers(SafeFileHandle handle);
} 

Here is a synchronous example of the same idea.

using System;
using System.IO;
using System.Threading;
using System.Runtime.InteropServices;
using System.Runtime.Remoting.Messaging;
using System.Security.Permissions;
using Microsoft.Win32.SafeHandles;

public class BulkImageProcSync
{
    public const String ImageBaseName = "tmpImage-";
    public const int numImages = 200;
    public const int numPixels = 512 * 512;

    // ProcessImage has a simple O(N) loop, and you can vary the number
    // of times you repeat that loop to make the application more CPU-
    // bound or more IO-bound.
    public static int processImageRepeats = 20;

    [SecurityPermissionAttribute(SecurityAction.Demand, Flags=SecurityPermissionFlag.UnmanagedCode)]
    public static void MakeImageFiles()
    {
        int sides = (int)Math.Sqrt(numPixels);
        Console.Write("Making {0} {1}x{1} images... ", numImages,
            sides);
        byte[] pixels = new byte[numPixels];
        int i;
        for (i = 0; i < numPixels; i++)
            pixels[i] = (byte)i;
        FileStream fs;
        for (i = 0; i < numImages; i++)
        {
            fs = new FileStream(ImageBaseName + i + ".tmp",
                FileMode.Create, FileAccess.Write, FileShare.None,
                8192, false);
            fs.Write(pixels, 0, pixels.Length);
            FlushFileBuffers(fs.SafeFileHandle);
            fs.Close();
        }
        fs = null;
        Console.WriteLine("Done.");
    }

    public static void ProcessImage(byte[] pixels, int imageNum)
    {
        Console.WriteLine("ProcessImage {0}", imageNum);
        int y;
        // Perform some CPU-intensive operation on the image.
        for (int x = 0; x < processImageRepeats; x += 1)
            for (y = 0; y < numPixels; y += 1)
                pixels[y] += 1;
        Console.WriteLine("ProcessImage {0} done.", imageNum);
    }

    public static void ProcessImagesInBulk()
    {
        Console.WriteLine("Processing images... ");
        long t0 = Environment.TickCount;
        byte[] pixels = new byte[numPixels];
        FileStream input;
        FileStream output;
        for (int i = 0; i < numImages; i++)
        {
            input = new FileStream(ImageBaseName + i + ".tmp",
                FileMode.Open, FileAccess.Read, FileShare.Read,
                4196, false);
            input.Read(pixels, 0, numPixels);
            input.Close();
            ProcessImage(pixels, i);
            output = new FileStream(ImageBaseName + i + ".done",
                FileMode.Create, FileAccess.Write, FileShare.None,
                4196, false);
            output.Write(pixels, 0, numPixels);
            output.Close();
        }
        input = null;
        output = null;
        long t1 = Environment.TickCount;
        Console.WriteLine("Total time processing images: {0}ms",
            (t1 - t0));
    }

    public static void Cleanup()
    {
        for (int i = 0; i < numImages; i++)
        {
            File.Delete(ImageBaseName + i + ".tmp");
            File.Delete(ImageBaseName + i + ".done");
        }
    }

    public static void TryToClearDiskCache()
    {
        byte[] bytes = new byte[100 * (1 << 20)];
        for (int i = 0; i < bytes.Length; i++)
            bytes[i] = 0;
        bytes = null;
        GC.Collect();
        Thread.Sleep(2000);
    }

    public static void Main(String[] args)
    {
        Console.WriteLine("Bulk image processing sample application," +
            " using synchronous I/O.");
        Console.WriteLine("Simulates applying a simple " +
            "transformation to {0} \"images.\"", numImages);
        Console.WriteLine("(ie, Sync FileStream benchmark).");
        Console.WriteLine("Warning - this test requires {0} " +
            "bytes of temporary space", (numPixels * numImages * 2));

        if (args.Length == 1)
        {
            processImageRepeats = Int32.Parse(args[0]);
            Console.WriteLine("ProcessImage inner loop � {0}",
                processImageRepeats);
        }

        MakeImageFiles();
        TryToClearDiskCache();
        ProcessImagesInBulk();
        Cleanup();
    }

    [DllImport("KERNEL32", SetLastError = true)]
    private static extern void FlushFileBuffers(SafeFileHandle handle);
}
 

Write Text to a File

using System;
using System.IO;
using System.Text;
using System.Collections.Generic;

class Program
{

    static void Main(string[] args)
    {

        string mydocpath = 
         Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
        StringBuilder sb = new StringBuilder();

        foreach (string txtName in Directory.EnumerateFiles(mydocpath,"*.txt"))
        {
            using (StreamReader sr = new StreamReader(txtName))
            {
                sb.AppendLine(txtName.ToString());
                sb.AppendLine("= = = = = =");
                sb.Append(sr.ReadToEnd());
                sb.AppendLine();
                sb.AppendLine();
            }

        }

        using (StreamWriter outfile = 
         new StreamWriter(mydocpath + @"\AllTxtFiles.txt"))
        {
            outfile.Write(sb.ToString());
        }
    }
} 

C++

using namespace System;
using namespace System::IO;
using namespace System::Text;
using namespace System::Collections::Generic;

ref class Program
{
public:
    static void Main()
    {
        String^ mydocpath =
         Environment::GetFolderPath(Environment::SpecialFolder::MyDocuments);
        StringBuilder^ sb = gcnew StringBuilder();

        for each (String^ txtName in Directory::EnumerateFiles(mydocpath, "*.txt"))
        {
            StreamReader^ sr = gcnew StreamReader(txtName);
            sb->AppendLine(txtName->ToString());
            sb->AppendLine("= = = = = =");
            sb->Append(sr->ReadToEnd());
            sb->AppendLine();
            sb->AppendLine();
            sr->Close();
        }

        StreamWriter^ outfile = gcnew StreamWriter(mydocpath + "\\AllTxtFiles.txt");
        outfile->Write(sb->ToString());
        outfile->Close();
    }
};

int main()
{
    Program::Main();
} 

How to: Read and Write to a Newly Created Data File

using System;
using System.IO;

class MyStream
{
    private const string FILE_NAME = "Test.data";

    public static void Main()
    {
        // Create the new, empty data file.
        if (File.Exists(FILE_NAME))
        {
            Console.WriteLine("{0} already exists!", FILE_NAME);
            return;
        }
        using (FileStream fs = new FileStream(FILE_NAME, FileMode.CreateNew))
        {
            // Create the writer for data.
            using (BinaryWriter w = new BinaryWriter(fs))
            {
                // Write data to Test.data.
                for (int i = 0; i < 11; i++)
                {
                    w.Write(i);
                }
            }
        }
        // Create the reader for data.
        using (FileStream fs = new FileStream(FILE_NAME, FileMode.Open, FileAccess.Read))
        {
            using (BinaryReader r = new BinaryReader(fs))
            {
                // Read data from Test.data.
                for (int i = 0; i < 11; i++)
                {
                    Console.WriteLine(r.ReadInt32());
                }
            }
        }
    }
} 

C++

using namespace System;
using namespace System::IO;

ref class MyStream
{
private:
    static String^ FILE_NAME = "Test.data";

public:
    static void Main()
    {
        // Create the new, empty data file.
        if (File::Exists(FILE_NAME))
        {
            Console::WriteLine("{0} already exists!", FILE_NAME);
            return;
        }
        FileStream^ fs = gcnew FileStream(FILE_NAME, FileMode::CreateNew);
        // Create the writer for data.
        BinaryWriter^ w = gcnew BinaryWriter(fs);
        // Write data to Test.data.
        for (int i = 0; i < 11; i++)
        {
            w->Write(i);
        }
        w->Close();
        fs->Close();
        // Create the reader for data.
        fs = gcnew FileStream(FILE_NAME, FileMode::Open, FileAccess::Read);
        BinaryReader^ r = gcnew BinaryReader(fs);
        // Read data from Test.data.
        for (int i = 0; i < 11; i++)
        {
            Console::WriteLine(r->ReadInt32());
        }
        fs->Close();
    }
};

int main()
{
    MyStream::Main();
}
 

Basic File I/O

The abstract base class Stream supports reading and writing bytes. Stream integrates asynchronous support. Its default implementations define synchronous reads and writes in terms of their corresponding asynchronous methods, and vice versa.
All classes that represent streams inherit from the Stream class. The Stream class and its derived classes provide a generic view of data sources and repositories, isolating the programmer from the specific details of the operating system and underlying devices.
Streams involve these fundamental operations:

• Streams can be read from. Reading is the transfer of data from a stream into a data structure, such as an array of bytes.
• Streams can be written to. Writing is the transfer of data from a data source into a stream.
• Streams can support seeking. Seeking is the querying and modifying of the current position within a stream.

Depending on the underlying data source or repository, streams might support only some of these capabilities. For example, NetworkStreams do not support seeking. The CanRead, CanWrite, and CanSeek properties of Stream and its derived classes determine the operations that various streams support.
For a list of common I/O tasks, see Common I/O Tasks.

Classes Used for File I/O

---

Directory provides static methods for creating, moving, and enumerating through directories and subdirectories. The DirectoryInfo class provides instance methods.
DirectoryInfo provides instance methods for creating, moving, and enumerating through directories and subdirectories. The Directory class provides static methods.
DriveInfo provides instance methods for accessing information about a drive.
File provides static methods for the creation, copying, deletion, moving, and opening of files, and aids in the creation of a FileStream. The FileInfo class provides instance methods.
FileInfo provides instance methods for the creation, copying, deletion, moving, and opening of files, and aids in the creation of a FileStream. The File class provides static methods.
FileStream supports random access to files through its Seek method. FileStream opens files synchronously by default, but supports asynchronous operation as well. File contains static methods, and FileInfo contains instance methods.
FileSystemInfo is the abstract base class for FileInfo and DirectoryInfo.
Path provides methods and properties for processing directory strings in a cross-platform manner.
DeflateStream provides methods and properties for compressing and decompressing streams using the Deflate algorithm.
GZipStream provides methods and properties for compressing and decompressing streams. By default, this class uses the same algorithm as the DeflateStream class, but can be extended to use other compression formats.
SerialPort provides methods and properties for controlling a serial port file resource.
File, FileInfo, DriveInfo, Path, Directory, and DirectoryInfo are sealed (in Microsoft Visual Basic, NotInheritable) classes. You can create new instances of these classes, but they cannot have derived classes.

Classes Used for Reading from and Writing to Streams

---

BinaryReader and BinaryWriter read and write encoded strings and primitive data types from and to Streams.
StreamReader reads characters from Streams, using Encoding to convert characters to and from bytes. StreamReader has a constructor that attempts to ascertain what the correct Encoding for a given Stream is, based on the presence of an Encoding-specific preamble, such as a byte order mark.
StreamWriter writes characters to Streams, using Encoding to convert characters to bytes.
StringReader reads characters from Strings. StringReader allows you to treat Strings with the same API, so your output can be either a Stream in any encoding or a String.
StringWriter writes characters to Strings. StringWriter allows you to treat Strings with the same API, so your output can be either a Stream in any encoding or a String.
TextReader is the abstract base class for StreamReader and StringReader. While the implementations of the abstract Stream class are designed for byte input and output, the implementations of TextReader are designed for Unicode character output.
TextWriter is the abstract base class for StreamWriter and StringWriter. While the implementations of the abstract Stream class are designed for byte input and output, the implementations of TextWriter are designed for Unicode character input.

Common I/O Stream Classes

---

A BufferedStream is a Stream that adds buffering to another Stream such as a NetworkStream. (FileStream already has buffering internally, and a MemoryStream does not need buffering.) A BufferedStream can be composed around some types of streams in order to improve read and write performance. A buffer is a block of bytes in memory used to cache data, thereby reducing the number of calls to the operating system.
A CryptoStream links data streams to cryptographic transformations. Although CryptoStream derives from Stream, it is not part of the System.IO namespace, but is in the System.Security.Cryptography namespace.
A MemoryStream is a nonbuffered stream whose encapsulated data is directly accessible in memory. This stream has no backing store and might be useful as a temporary buffer.
A NetworkStream represents a Stream over a network connection. Although NetworkStream derives from Stream, it is not part of the System.IO namespace, but is in the System.Net.Sockets namespace.

I/O and Security

---

When using the classes in the System.IO namespace, operating system security requirements such as access control lists (ACLs) must be satisfied for access to be allowed. This requirement is in addition to any FileIOPermission requirements.