Main Page

Previous Next

Memory-Mapped Files

A memory-mapped file is a file that has its contents mapped into an area of virtual memory in your computer so you can reference or update the data directly without performing any explicit file read or write operations on the physical file yourself. The memory that a file maps to may be paged in or out by the operating system, just like any other memory in your computer, so its immediate availability in real memory is not guaranteed. Because of the potentially immediate availability of the data it contains, a memory-mapped file is particularly useful when you need to access the file randomly. Your program code can reference data in the file as though it were resident in memory.

The map() method for a FileChannel object will return a reference to a buffer of type MappedByteBuffer that will map to a specified part of the channel's file: :

map(int mode, long position, long size)

Maps a region of the channel's file to a buffer of type MappedByteBuffer. The file region that is mapped starts at position in the file and is of length size bytes. The first argument, mode, specifies how the buffer's memory may be accessed and can be any of the following three constant values, defined in the FileChannel class:

MAP_RO. This is valid if the channel was opened for reading the file, in other words, if the channel was obtained from a FileInputStream object or a RandomAccessFile object. In this mode the buffer is read-only. If you try to modify the buffer's contents a ReadOnlyBufferException will be thrown.

MAP_RW. This is valid if the channel was obtained from a RandomAccessFile object with "rw" as its access mode. You can access and change the contents of the buffer and any changes to the contents will eventually be propagated to the file.

MAP_COW. The COW part of the name is for Copy On Write. This option for mode is also only valid if the channel was obtained from a RandomAccessFile object with "rw" as its access mode. You can access or change the buffer but changes will not be propagated to the file. Private copies of modified portions of the buffer will be created and used for subsequent buffer accesses.

Because the MappedByteBuffer class is a subclass of the ByteBuffer class, you have all the ByteBuffer methods available for a MappedByteBuffer object. This implies that you can create view buffers for a MappedByteBuffer object, for instance.

The MappedByteBuffer class defines three methods of its own to add to those inherited from the ByteBuffer class:

force()

If the buffer was mapped in MAP_RW mode this method forces any changes made to the buffer's contents to be written to the file and returns a reference to the buffer. For buffers created with MAP_RO or MAP_COW mode this method has no effect.

load()

Loads the contents of the buffer into memory and returns a reference to the buffer.

isLoaded()

Returns true if this buffer's contents is available in physical memory and false otherwise.

Using a memory-mapped file through a MappedByteBuffer is simplicity itself, so let's try it.

Try It Out – Using a Memory-Mapped File

We will access and modify the primes.bin file using a MappedByteBuffer. Here's the code:

import java.io.*;
import java.nio.*;
import java.nio.channels.FileChannel;

public class MemoryMappedFile {
  public static void main(String[] args) {
    File aFile = new File("C:/Beg Java Stuff/primes.bin");
    RandomAccessFile ioFile = null;
    
    try {
      ioFile = new RandomAccessFile(aFile,"rw");

    } catch(FileNotFoundException e) {
      e.printStackTrace(System.err);
      System.exit(1);
    }
    FileChannel ioChannel = ioFile.getChannel();
    
    
    final int PRIMESREQUIRED = 10;
    long[] primes = new long[PRIMESREQUIRED];

    int index = 0;                       // Position for a prime in the file
    final long REPLACEMENT = 999999L;    // Replacement for a selected prime

    try {
      final int PRIMECOUNT = (int)ioChannel.size()/8;
      MappedByteBuffer buf =
              ioChannel.map(ioChannel.MAP_RW, 0L, ioChannel.size()).load();

      for(int i = 0 ; i<PRIMESREQUIRED ; i++) {
        index = 8*(int)(PRIMECOUNT*Math.random());    
        primes[i] = buf.getLong(index);
        buf.putLong(index, REPLACEMENT );
      }
      StringBuffer str = null;        
      for(int i = 0 ; i<PRIMESREQUIRED ; i++) {
        str = new StringBuffer("           ").append(primes[i]);          
          System.out.print((i%5 == 0 ? "\n" : "") + str.substring(str.length()-12,
                           str.length()));
      }
      ioFile.close();                   // Close the file and the channel
 
    } catch(IOException e) {
      e.printStackTrace(System.err);
      System.exit(1);
    }
      System.exit(0);
  }
}

This should output ten randomly selected primes but some or all of the selections may turn out to be 999999L, the value of REPLACEMENT.

How It Works

The statements of interest are those that are shaded. The others we have seen before. We create a RandomAccessFile object with "rw" access to the file:

ioFile = new RandomAccessFile(aFile,"rw");

This statement executes in a try block since it can throw a FileNotFoundException. We get the file channel for ioFile with the statement:

FileChannel ioChannel = ioFile.getChannel();

We then create and load a MappedByteBuffer object with the statement:

MappedByteBuffer buf = 
          ioChannel.map(ioChannel.MAP_RW, 0L, ioChannel.size()).load();

The buffer is created with the mode that permits the buffer to be accessed or modified and modifications will be written to the file. The buffer maps to the entire file since we specify the start file position as zero and the length mapped as the length of the file. The map() method returns a reference to the MappedByteBuffer object that is created and we use this to call its load() method to request that the contents of the file are loaded into memory immediately. The load() method also returns the same buffer reference that is stored in buf.

Note that it is not essential to call the load method before you access the data in the buffer. If the data is not available when you try to access it through the MappedByteBuffer object, it will be loaded for you. Try running the example with the call to load() removed. It should work the same as before.

Inside the for loop, we retrieve a value from the buffer at a random position, index:

primes[i] = buf.getLong(index);

Note that we have not needed to execute any channel read() operations. The file contents are available directly through the buffer.

Next we change the value at the position we retrieved the value from to store in primes[i]:

buf.putLong(index, REPLACEMENT );

This will change the contents of the buffer and this change will subsequently be written to the file at some point. When this occurs depends on the underlying operating system.

Finally we output the contents of the primes array. We have been able to access and modify the contents of the file without having to execute any explicit I/O operations on the file. This is potentially much faster than using explicit read and write operations. How much faster depends on how your operating system handles memory-mapped files.

There is one risky aspect to memory-mapped files that we need to consider.

Locking a File

You need to take care that an external program does not modify a memory-mapped file, especially if the file could be truncated externally while you are accessing it. If you try to access a part of the file through a MappedByteBuffer that has become inaccessible because a segment has been chopped off the end of the file, then the results are somewhat unpredictable. You may get a junk value back that your program may not recognize as such, or an exception of some kind may be thrown. You can acquire a lock on the file to prevent this sort of problem. A file lock simply ensures your right of access to the file and may also inhibit the ability of others to access the file as long as your lock is in effect. This facility will only be available if the underlying operating system supports file locking.

A lock on a file is encapsulated by an object of the FileLock class that is defined in the java.nio.channels package. The lock() method for a FileChannel object tries to obtain an exclusive lock on the channel's file. Acquiring an exclusive lock on a file ensures that another program cannot access the file. Here's one way to obtain an exclusive lock on a file:

FileLock ioFileLock = null;
try {
  ioFileLock = ioChannel.lock(); 

} catch (IOException e) {
  e.printStackTrace(System.err);
  System.exit(1);
}

This method will attempt to acquire an exclusive lock on the channel's file so that no other program or thread can access the file while this channel holds the lock. If another program or thread already has a lock on the file, the lock() method will block until the lock on the file is released and can be acquired by this channel. The lock that is acquired is owned by the channel, ioChannel, and will be automatically released when the channel is closed. By saving the reference to the FileLock object, we can release the lock on the file when we are done by calling the release() method for the FileLock object. This invalidates the lock so file access is no longer restricted. You can call the isValid() method for a FileLock object to determine whether it is valid. A return value of true indicates a valid lock, otherwise false will be returned. Note that once created, a FileLock object is immutable. It also has no further effect on file access once it has been invalidated. If you want to lock the file a second time you must acquire a new lock.

Having your program hang until a lock is acquired is not an ideal situation. It is quite possible a file could be locked permanently – at least until the computer is rebooted – because of a programming error in another program, in which case your program will hang indefinitely. The tryLock() method for a channel offers an alternative way of requesting a lock that does not block. It either returns a reference to a valid FileLock object or returns null if the lock could not be acquired. This gives your program a chance to do something else or retire gracefully:

FileLock ioFileLock = null;
try {
  ioFileLock = ioChannel.tryLock(); 
  if(ioFileLock == null) {
    System.out.println("The file's locked – again!! Oh, I give up...");
    System.exit(1);
  }
} catch (IOException e) {
  e.printStackTrace(System.err);
  System.exit(1);
}

We will see in an example a better response to a lock than this, but you get the idea.

Locking Part of a File

There are overloaded versions of the lock() and tryLock() methods that allow you to specify just a part of the file you want to obtain a lock on:

lock(long position, long size, boolean shared)

Requests a lock on the region of this channel's file starting at position and of length size. If the last argument is true, the lock requested is a shared lock. If it is false the lock requested is an exclusive lock. If the lock cannot be obtained for any reason, the method will block until the lock can be obtained or the channel is closed by another thread.

tryLock(long position, long size, boolean shared)

This works in the same way as the method above, except that null will be returned if the requested lock cannot be acquired. This avoids the potential for hanging your program indefinitely.

The effect of a shared lock is to prevent an exclusive lock being acquired by another program that overlaps the region that is locked. However, a shared lock does permit another program to acquire a shared lock on a region of the file that may overlap the region to which the original shared lock applies. This implies that more than one program may be accessing and updating the same region of the file, so the effect of a shared lock is simply to ensure your code is not prevented from doing whatever it is doing by some other program with a shared lock on the file. Some operating systems do not support shared locks, in which case the request will always be treated as an exclusive lock.

Note that a single virtual machine does not allow overlapping locks so different threads running on the same VM cannot have overlapping locks on a file. However, the locks within two or more JVMs on the same computer can overlap. If another program changing the data in a file would cause a problem for you then the safe thing to do is to obtain an exclusive lock on the region of the file you are working with.

Practical File Locking Considerations

You can apply file locks in any context, not just with memory-mapped files. The fact that all or part of a file can be locked by a program means that you cannot ignore file locking when you are writing a real-world Java application that may execute in a context where file locking is supported. You need to include at least shared file locks for regions of a file that your program uses. In most instances, though, you will want to use exclusive locks since external changes to a file's contents are usually a problem if you are accessing the same data.

You don't need to obtain an exclusive lock on an entire file. Generally, if it is likely that other programs will be using the same file concurrently, it is not reasonable practice to lock everyone else out, unless it is absolutely necessary, such as a situation in which you may be performing a transacted operation that must either succeed or fail entirely. Circumstances where it would be necessary are when the correctness of your program result is dependent on the entire file's contents not changing. If you were computing a checksum for a file for instance, you need to lock the entire file. Any changes while your checksum calculation is in progress are likely to make it incorrect.

Most of the time it is quite sufficient to lock the portion of the file you are working with, and then release it once you have done with it. We can show the idea in the context of the program that lists the primes from the primes.bin file.

Try It Out – Using a File Lock

We will lock the region of the primes.bin file that we intend to read, and then release it after the read operation is complete. We will use the tryLock() method, since it does not block, and try to acquire the lock again if it fails to return a reference to a FileLock object. To do this sensibly we need to be able to pause the current thread rather than roaring round a tight loop frantically calling the tryLock() method. We will bring forward a capability from Chapter 15 to do this for us. We can pause the current thread by 200 milliseconds with the following code:

try {
  Thread.sleep(200);    // Wait for 200 milliseconds

} catch(InterruptedException e) {
  e.printStackTrace(System.err);
}

The static sleep() method in the Thread class causes the current thread to sleep for the number of milliseconds specified by the argument. While our current thread is sleeping other threads can execute, so whoever has a lock on our file has a chance to release it.

Here's the code for the complete example:

import java.io.*;
import java.nio.*;
import java.nio.channels.*;                 // For FileChannel and FileLock

public class LockingPrimesRead {
  public static void main(String[] args) 
    File aFile = new File("C:/Beg Java Stuff/primes.bin");
    FileInputStream inFile = null;
   
    try {
      inFile = new FileInputStream(aFile); 

    } catch(FileNotFoundException e) {
      e.printStackTrace(System.err);
      System.exit(1);
    }
    FileChannel inChannel = inFile.getChannel();
    final int PRIMECOUNT = 6;
    ByteBuffer buf = ByteBuffer.allocate(8*PRIMECOUNT);  
    long[] primes = new long[PRIMECOUNT];
    try {
      int primesRead = 0;
      FileLock inLock = null;
      while(true) {
        int tryLockCount = 0;

        // Get a lock on the file region we want to read
        while(true) {
          inLock = inChannel.tryLock(inChannel.position(), 
                                     buf.remaining(), 
                                     false);
          if(inLock == null) {
            if(++tryLockCount <100)
            {
              try {
                Thread.sleep(200);                    // Wait for 200 milliseconds

              } catch(InterruptedException e) {
                e.printStackTrace(System.err);
              }
              continue;                     // Continue with next loop iteration

            } else {
              System.out.println("Failed to acquire lock after "
                               + tryLockCount+" tries."+ " Terminating...");
              System.exit(1);
            } else {
              System.out.println("\nAcquired file lock.");
              break;
            }
          }
        }

        // Now read the file
        if(inChannel.read(buf) == -1)
          break;
        inLock.release();
        System.out.println("Released file lock.");
          
        LongBuffer longBuf = ((ByteBuffer)(buf.flip())).asLongBuffer();
        primesRead = longBuf.remaining();
          longBuf.get(primes,0, longBuf.remaining());
        StringBuffer str = null;        
        for(int i = 0 ; i< primesRead ; i++) {
          if(i%6 == 0)
              System.out.println();
          str = new StringBuffer("           ").append(primes[i]);          
          System.out.print(str.substring(str.length()-12));
        }
        buf.clear();                    // Clear the buffer for the next read
      }

      System.out.println("\nEOF reached.");
      inFile.close();                   // Close the file and the channel

    } catch(IOException e) {
      e.printStackTrace(System.err);
      System.exit(1);
    }
      System.exit(0);
  }
}

This will output primes from the file the same as the ReadPrimes example does, but interspersed with comments showing where we acquire and release the file lock.

How It Works

The overall while loop for reading the file is now indefinite since we need to obtain a file lock before reading the file. We attempt to acquire the file lock in the inner while loop with the statement:

inLock = inChannel.tryLock(inChannel.position(), buf.remaining(), false);

This requests an exclusive lock on buf.remaining() bytes in the file starting with the byte at the current file position. Acquiring a lock on just the part of the file that we want to read ensures that other programs are not prevented from accessing the rest of the file. We have to test the value returned by the tryLock() method for null to determine whether we have obtained a lock or not. The if statement that does this looks quite complex, but its overall operation is quite simple:

if(inLock == null)
  // Cause the thread to sleep for a while before retrying the lock...
else
  // We have the lock so break out of the loop...

If inLock is null, we try to acquire the lock again with the following code:

if(++tryLockCount <100) {
  try {
    Thread.sleep(200);                 // Wait for 200 milliseconds

  } catch(InterruptedException e) {
    e.printStackTrace(System.err);
  }
  continue;                           // Continue with next loop iteration
} else {
  System.out.println("Failed to acquire lock after "+tryLockCount+" tries."
                   + " Terminating...");
  System.exit(1);
}

If we have tried fewer than 100 times to acquire the lock, we have another go after sending the current thread to sleep for 200 milliseconds. Once we have failed to acquire a lock 100 times, we abandon reading the file and exit the program.

Once we have acquired a lock, we read the file in the usual way and release the lock:

if(inChannel.read(buf) == -1)
  break;
inLock.release();
System.out.println("Released file lock.");

By releasing the lock immediately after reading the file, we ensure that the amount of time the file is blocked is a minimum. Of course, if the read() method returns -1 because EOF has been reached, we won't call the release() method for the FileLock object here because we exit the outer loop. However, after exiting the outer while loop we close the file stream and the channel, and closing the channel will release the lock.

Previous Next
JavaScript Editor Java Tutorials Free JavaScript Editor