Files in a programming sense are really not very different from files that you use in a word processor or other application: you open them, do some work and then close them again. The biggest differences are that in a program you access the file sequentially, that is, you read one line at a time starting at the beginning. (In practice the word processor often does the same, it just holds the entire file in memory while you work on it and then writes it all back out when you close it.) The other difference is that, when programming, you normally open the file as read only or write only. You can write by creating a new file from scratch (or overwriting an existing one) or by appending to an existing one.

One other thing you can do while processing a file is that you can go back to the beginning.

Files - Input and Output

Let's see that in practice. We will assume that a file exists called menu.txt and that it holds a list of meals:

spam & eggs
spam & chips
spam & spam

Now we will write a program to read the file and display the output - like the 'cat' command in Unix or the 'type' command in Windows CMD shells.

# First open the file to read(r)
inp = open("menu.txt","r")
# read the file line by line
for line in inp:
    print( line )
# Now close it again
inp.close()

Note 1: open() takes two arguments. The first is the filename (which may be passed as a variable or a literal string, as we did here). The second is the mode. The mode determines whether we are opening the file for reading(r) or writing(w), and also whether it's for text or binary usage - by adding a 'b' to the 'r' or 'w', as in:

open(fn,"rb")

Note 2: We read the file in a for loop. Recall that Python's for loop acts as a foreach loop on a collection. It returns each element of the collection. A text file can be considered a collection of lines and so the loop reads each line in turn.

Note 3: We close the file using a function preceded by the file variable. This notation is known as method invocation as described in the Modules and Functions topic. You can, if it helps, think of a file variable as being a reference to a module containing functions that operate on files and which we automatically import every time we create a file type variable.

In Python, files are automatically closed at the end of the program but it is good practice to get into the habit of closing your files explicitly. Why? Well, the operating system may not write the data out to the file until it is closed (this can boost performance). What this means is that if the program exits unexpectedly there is a danger that your precious data may not have been written to the file! So the moral is: once you finish writing to a file, close it.

Note 4: We have not specified the full path to the file in the code above so the file will be treated as being in the current folder. However, we can pass a full path name to open() instead of just the file name. There is a wrinkle when using Windows however, because the \ character used to separate folders in a Windows path has a special meaning inside a Python string. So, when specifying paths in Python it is best to always use the / character instead and that will work on any Operating System including Windows.(There is more information in a box below)

Now, consider how you could cope with long files. You couldn't display all of the file on a single screen so we need to pause after each screenful of text. You might use a line_count variable which is incremented for each line and then tested to see whether it is equal to 25 (for a 25 line screen). If so, you request the user to press a key (enter, say) before resetting line_count to zero and continuing. You might like to try that as an exercise...

Another way of reading a file is to use a while loop and a method of the file object called readline(). The advantage of this is that we can stop processing the file as soon as we find the data we need, this can greatly speed things up if processing long files. However, it is a little bit more complex, so let's look at the previous example again using a while loop:

# First open the file to read(r)
inp = open("menu.txt","r")
# read the file and print each line
while True:
    line = inp.readline()
    if not line: break
    print( line )
# Now close it again
inp.close()

Note: we use the break technique mentioned in the branching topic to exit the loop if the line is empty (recall an empty line counts as false in Boolean terms). Thereafter we printed each line and went around the loop again. Finally, after exiting the while loop, we closed the file. If we wanted to stop at a certain point in the file we would have introduced a branch condition inside the while loop and if it detected the stop condition we simply call break there too so that the loop will terminate.

Really that's all there is to it. You open the file, read it in and manipulate it any way you want to. When you're finished you close the file. However there is one little niggle you may have noticed in the previous example: the lines read from the file have a newline character at the end, so you wind up with blank lines using print() (which adds its own newline). To avoid that Python provides a string method called strip() which will remove whitespace, or non-printable characters, from both ends of a string. (It also has cousins, called rstrip and lstrip, which can from strip one end only) If we substitute the print() line above with:

    print( line.rstrip() )  #only strip right hand end

Everything should now work just fine.

To create a 'copy' command in Python, we simply open a new file in write mode and write the lines to that file instead of printing them. Like this:

# Create the equivalent of: COPY MENU.TXT MENU.BAK

# First open the files to read(r) and write(w)
inp = open("menu.txt","r")
outp = open("menu.bak","w")

# read file, copying each line to new file
for line in inp:
    outp.write(line)

print( "1 file copied..." )

# Now close the files
inp.close()
outp.close()

Did you notice that I added a print() statement at the end, just to reassure the user that something actually happened? This kind of user feedback is usually a good idea.

Because we wrote out the same line that we read in there was no problems with newline characters here. But if we had been writing out strings which we created, or which we had stripped earlier we would have needed to add a newline to the end of the output string, like this:

outp.write(line + '\n') # \n is a newline

Let's look at how we might incorporate that into our copy program. Instead of simply copying the menu we will add todays date to the top. That way we can easily generate a daily menu from the easily modified text file of meals. All we need to do is write out a couple of lines at the top of the new file before copying the menu.txt file, like this:

import time
# Create daily menu based on MENU.TXT
# First open the files to read(r) and write(w)
inp = open("menu.txt","r")
outp = open("menu.prn","w")

# Create todays date string 
today = time.localtime(time.time())
theDate = time.strftime("%A %B %d", today)

# Add Banner text and a blank line
outp.write("Menu for %s\n\n" % theDate) 

# copy each line of menu.txt to new file
for line in inp:
    outp.write(line)

print( "Menu created for %s..." % theDate )

# Now close the files
inp.close()
outp.close()

Note that we use the time module to get todays date (time.time()) and convert it into a tuple of values (time.localtime()) which are then used by time.strftime() (check the documentation for time.strftime to see what else it can do) to produce a string which, when inserted into a title message using string formatting, looks like:

Menu for Sunday September 19

Spam & Eggs
Spam &...

Although we added two '\n' characters at the end of the string there is only one blank line printed, that's because one of them is the newline at the end of the title itself. Managing the creation and removal of newline characters is one of the more irritating aspects of handling text files.

>>> f = open('C:\test.dat')

>>> f = open(r'C:\test.dat')
>>> f = open('C:/test.dat')
>>> f = open('C:\\test.dat')

Appending data

One final twist in file processing is that you might want to append data to the end of an existing file. One way to do that would be to open the file for input, read the data into a list, append the data to the list and then write the whole list out to a new version of the old file. If the file is short that's not a problem but if the file is very large, maybe over 100Mb, then you could run out of memory to hold the list, and it would take quite a long time. Fortunately there's another mode "a" that we can pass to open() which allows us to append directly to an existing file just by writing. Even better, if the file doesn't exist it will open a new file just as if you'd specified "w".

As an example, let's assume we have a log file that we use for capturing error messages. We don't want to delete the existing messages so we choose to append the error, like this:

def logError(msg):
   err = open("Errors.log","a")
   err.write(msg)
   err.close()

In the real world we would probably want to limit the size of the file in some way. A common technique is to create a filename based on the date, thus when the date changes we automatically create a new file and it is easy for the maintainers of the system to find the errors for a particular day and to archive away old error files if they are not needed. (Remember, from the menu example above, that the time module can be used to find out the current date.)

with open('Errors.log',"r") as inp:
    for line in inp:
        print( line )                           

You remember the address book program we introduced during the Raw Materials topic and then expanded in the Talking to the User topic? Let's start to make it really useful by saving it to a file and, of course, reading the file at start-up. We'll do this by writing some functions. So in this example we pull together several of the strands that we've covered in the last few topics.

The basic design will require a function to read the file at start-up and another to write the file at the end of the program. We will also create a function to present the user with a menu of options and a separate function for each menu selection. The menu will allow the user to:

• Add an entry to the address book
• Remove an entry from the book
• Find and display an existing entry
• Quit the program

Loading the Address Book

import os
filename = "addbook.dat"

def readBook(book):
    if os.path.exists(filename):
       with open(filename,'r') as store:
          for line in store:
             name = line.rstrip()
             entry = next(store).rstrip()
             book[name] = entry

Note 1: We import the os module which we use to check that the file path actually exists before opening the file.

Note 2: We defined the filename as a module level variable so we can use it both in loading and saving the data.

Note 3: We use rstrip() to remove the new-line character from the end of the line. Also notice the next() funtion to fetch the next line from the file within the loop. This effectively means we are reading two lines at a time as we progress through the loop.

Saving the Address Book

def saveBook(book):
    with open(filename, 'w') as store:
       for name,entry in book.items():
         store.write(name + '\n')
         store.write(entry + '\n')

Notice we need to add a newline character ('\n') when we write the data. Also note that we write two lines for each entry, this mirrors the fact that we processed two lines when reading the file.

Getting User Input

def getChoice(menu, length):
    print( menu )
    prompt = "Select a choice(1-%d): "  % length
    choice = int( input(prompt) )
    return choice

Note: We receive a length parameter which tells us how many menu entries there are. This allows us to create a prompt that specifies the correct number range.

Adding an Entry

def addEntry(book):
    name =  input("Enter a name: ")
    entry = input("Enter street, town and phone number: ")
    book[name] = entry

Removing an entry

def removeEntry(book):
    name = input("Enter a name: ")
    del(book[name])

Finding an entry

def findEntry(book):
    name = input("Enter a name: ")
    if name in book:
       print( name, book[name] )
    else: print( "Sorry, no entry for: ", name )

Quitting the program

Actually I won't write a separate function for this, instead I'll make the quit option the test in my menu while loop. So the main program will look like this:

def main():
    theMenu = '''
    1) Add Entry
    2) Remove Entry
    3) Find Entry
    4) Quit and save
    '''
    theBook = {}
    readBook(theBook)
    while True:
        choice = getChoice(theMenu, 4)
        if choice == 4: break
        
        if choice == 1:
            addEntry(theBook)
        elif choice == 2:
            removeEntry(theBook)
        elif choice == 3:
            findEntry(theBook)
        else: print( "Invalid choice, try again" )
    saveBook(theBook)

Now the only thing left to do is call the main() function when the program is run, and to do that we use a bit of Python magic like this:

if __name__ == "__main__":
    main()

This mysterious bit of code allows us to use any python file as a module by importing it, or as a program by running it. The difference is that when the program is imported, Python sets the internal variable __name__ to the module name but when the file is run as a program, the value of __name__ is set to "__main__". This means the main() function only gets called if the file is run as a program, but not when the file is imported. Sneaky, eh?

Now if you type all that code into a new text file and save it as addressbook.py, you should be able to run it from an OS prompt by typing:

C:\PROJECTS> python addressbook.py

Or just double click the file in Windows Explorer and it should start up in its own CMD window, and the window will close when you select the quit option.

Or, in Linux:

$ python addressbook.py

This 60 odd line program is typical of the sort of thing you can start writing for yourself. There are a couple of things we can do to improve it which I'll cover in the next section, but even as it stands it's a reasonably useful little tool.

VBScript and JavaScript

Neither VBScript nor JavaScript have native file handling capabilities. This is a security feature to ensure nobody can read your files when you innocently load a web page, but it does restrict their general usefulness. However, as we saw with reusable modules there is a way to do it using Windows Script Host. WSH provides a FileSystem object which allows any WSH language to read files. We will look at a JavaScript example in detail then show similar code in VBScript for comparison, but as before the key elements will really be calls to the WScript objects.

Before we can look at the code in detail it's worth taking time to describe the FileSystem Object Model. An Object Model is a set of related objects which can be used by the programmer. The WSH FileSystem object model consists of the FSO object, a number of File objects, including the TextFile object which we will use. There are also some helper objects, most notable of which is, for our purposes, the TextStream object. Basically we will create an instance of the FSO object, then use it to create our TextFile objects and from these in turn create TextStream objects to which we can read or write text. The TextStream objects themselves are what we actually read/write from the files.

Type the following code into a file called testFiles.js and run it using cscript as described in the earlier introduction to WSH.

Opening a file

To open a file in WSH we create an FSO object then create a TextFile object from that:

var fileName, fso, txtFile, outFile, line;

// Get file name
fso = new ActiveXObject("Scripting.FileSystemObject");
WScript.Echo("What file name? ");
fileName = WScript.StdIn.Readline();

// open inFile to read, outFile to write
inFile = fso.OpenTextFile(fileName, 1); // mode 1 = Read
fileName = fileName + ".BAK"
outFile = fso.CreateTextFile(fileName);

Reading and Writing a file

// loop over file till it reaches the end
while ( !inFile.AtEndOfStream ){
    line = inFile.ReadLine();
    WScript.Echo(line);
    outFile.WriteLine( line );
    }

Closing files

inFile.close();
outFile.close();

And in VBScript

Save the following as testFIles.ws and then run it using:

cscript testfiles.ws

Or alternatively, put the bit between the script tags into a file called testFile.vbs and run that instead. The .ws format allows you to mix JavaScript and VBScript code in the same file by simply using multiple script tags, should you want to...

<?xml version="1.0"?>

<job>
  <script type="text/vbscript">
      Dim fso, inFile, outFile, inFileName, outFileName
      Set fso = CreateObject("Scripting.FileSystemObject")
      
      WScript.Echo "Type a filename to backup"
      inFileName = WScript.StdIn.ReadLine
      outFileName = inFileName &amp; ".BAK"
      
      ' open the files
      Set inFile = fso.OpenTextFile(inFileName, 1)
      Set outFile = fso.CreateTextFile(outFileName)

      ' read the file and write to the backup copy
      Do While not inFile.AtEndOfStream
         line = inFile.ReadLine
         outFile.WriteLine(line)
      Loop
      
      ' close both files
      inFile.Close
      outFile.Close
      
      WScript.Echo inFileName &amp; " backed up to " &amp; outFileName
  </script>
</job>

Handling Non-Text Files

Handling text is one of the most common things that programmers do, but sometimes we need to process raw binary data too. This is very rarely done in VBScript or JavaScript so I will only be covering how Python does it.

Opening and Closing Binary Files

The key difference between text files and binary files is that text files are composed of octets, or bytes, of binary data whereby each byte represents a character and the end of the file is marked by a special byte pattern, known generically as end of file, or eof. A binary file contains arbitrary binary data and thus no specific value can be used to identify end of file, thus a different mode of operation is required to read these files. The end result of this is that when we open a binary file in Python (or indeed any other language) we must specify that it is being opened in binary mode or risk the data being read being truncated at the first eof character that Python finds in the data. The way we do this in Python is to add a 'b' to the mode parameter, like this:

binfile = open("aBinaryFile.bin","rb")

The only difference from opening a text file is the mode value of "rb". You can use any of the other modes too, simply add a 'b': "wb" to write, "ab" to append.

Closing a binary file is no different to a text file, simply call the close() method of the open file object:

binfile.close()

Because the file was opened in binary mode there is no need to give Python any extra information, it knows how to close the file correctly.

The struct Module

To encode/decode binary data Python provides a module called struct, short for structure. struct works very much like the format strings we have been using to print mixed data. We provide a string representing the data we are reading and apply it to the byte stream that we are trying to interpret. We can also use struct to convert a set of data to a byte stream for writing, either to a binary file (or even a communications line!).

There are many different conversion format codes but we will only use the integer and string codes here. (You can look up the others on the Python documentation for the struct module.) The codes for integer and string are i, and s respectively. The struct format strings consist of sequences of codes with numbers pre-pended to indicate how many of the items we need. The exception is the s code where the prepended number means the length of the string. For example 4s means a string of four characters (note 4 characters not 4 strings!).

Let's assume we wanted to write the address details, from our Address Book program above, as binary data with the street number as an integer and the rest as a string (This is a bad idea in practice since street "numbers" sometimes include letters!). The format string would look like:

'i34s' # assuming 34 characters in the address!

To cope with multiple address lengths we could write a function to create the binary string like this:

def formatAddress(address): 
    # split breaks a string into a list of 'words'
    fields = address.split()
    number = int(fields[0])
    rest = bytes(' '.join(fields[1:],'utf8') # create a byte string
    format = "i%ds" % len(rest) # create the format string
    return struct.pack(format, number, rest)

So we used a string method - split() - (more on them in the next topic!) to split the address string into its parts, extract the first one as the number and then use another string method, join to join the remaining fields back together separated by spaces. We also need to convert the string into a bytes array because that's what the struct module uses. The length of that string is the number we need in the struct format string so we use the len() function in conjunction with a normal format string to build a struct format string. Phew!

formatAddress() will return a sequence of bytes containing the binary representation of our address. Now that we have our binary data let's see how we can write that to a binary file and then read it back again.

Reading & Writing Using struct

Let's create a binary file containing a single address line using the formatAddress() function defined above. We need to open the file for writing in 'wb' mode, encode the data, write it to the file and then close the file. Let's try it:

import struct
f = open('address.bin','wb')
data = "10 Some St, Anytown, 0171 234 8765"
bindata = formatAddress(data)
print( "Binary data before saving: ", repr(bindata) )
f.write(bindata)
f.close()

You can check that the data is indeed in binary format by opening address.bin in notepad. The characters will be readable but the number will not look like 10! In fact it has disappeared! If you have an editor which can read binary files (e.g vim or emacs) and use that to open address.bin you will see that the start of the file has 4 bytes. The first of these may look like a newline character and the rest are zeros. Now it turns out that, just coincidentally, the numerical value of newline is 10! As we can show using Python:

>>> ord('\n')
10

The ord() function simply returns the numeric value of a given character. So the first 4 bytes are 10,0,0,0 in decimal (or 0xA,0x0,0x0,0x0 in hexadecimal, the system usually used to display binary data - since it is much more concise than using pure binary).

On a 32 bit computer an integer takes up 4 bytes. So the integer value '10' has been converted by the struct module into the 4 byte sequence 10, 0, 0, 0. Now on intel micro-processors the byte sequence is to put the least significant byte first so that, reading it in reverse, gives us the true "binary" value: 0,0,0,10.

Which is the integer value 10 expressed as 4 decimal bytes. The rest of the data is basically the original text string and so appears in its normal character format.

Be sure not to save the file from within Notepad since although Notepad can load some binary files it cannot save them as binary, it will try to convert the binary to text and can corrupt the data in the process! It is worth pointing out here that the file extension .bin that we used is purely for our convenience, it has no bearing on whether the file is binary or text format. Some Operating Systems use the extension to determine what programme they will use to open the file, but you can change the extension by simply renaming the file, the content will not change it will still be binary or text whichever it was originally. (You can prove this by renaming a text file in Windows to .exe whereupon Windows will treat the file as an executable, but when you try to run it you will get an error because the text is not really executable binary code! If you now rename it back to .txt the file will open in Notepad exactly as it did before, the content has not been altered at all - in fact you could even have opened the text in Notepad while it was named as a .exe and it would have worked just as well!)

To read our binary data back again we need to open the file in 'rb' mode, read the data into a sequence of bytes, close the file and finally unpack the data using a struct format string. The question is: how do we tell what the format string looks like? In general we would need to find the binary format from the file definition (there are several web sites which provide this information - for example Adobe publish the definition of their common PDF binary format). In our case we know it must be like the one we created in formatAddress(), namely 'iNs' where N is a variable number. How do we determine the value of N?

The struct module provides some helper functions that return the size of each data type, so by firing up the Python prompt and experimenting we can find out how many bytes of data we will get back for each data type:

>>> import struct
>>> print struct.calcsize('i')
4
>>> print struct.calcsize('s')
1

Ok, we know that our data will comprise 4 bytes for the number and one byte for each character. So N will be the total length of the data minus 4. Let's try using that to read our file:

import struct
f = open('address.bin','rb')
data = f.read()
f.close()

fmtString = "i%ds" % (len(data) - 4)
number, rest = struct.unpack(fmtString, data)
rest = rest.decode('utf8') #convert bytes to string
address = ' '.join((str(number),rest))

print( "Address after restoring data:", address )

Note: We had to convert rest to a string using the decode() function since Python considered it to be of type bytes (see the sidebar above) which won't work with join().

And that's it on binary data files, or at least as much as I'm going to say on the subject. As you can see using binary data introduces several complications and unless you have a very good reason I don't recommend it. But at least if you do need to read a binary file, you can do it (provided you know what the data represented in the first place of course!)

Random Access to Files

The last aspect of file handling that I'll consider is called random access. Random access means moving directly to a particular part of the file without reading all the intervening data. Some programming languages provide a special indexed file type that can do this very quickly but in most languages its built on top of the normal sequential file access that we have been using up till now.

The concept used is that of a cursor that marks the current position within the file, literally how many bytes we are from the beginning. We can move this cursor relative to its current position or relative to the start of the file. We can also ask the file to tell us where the cursor is currently.

By using a fixed linelength (perhaps by padding our data strings with spaces or some other character where necessary) we can jump to the start of a particular line by multiplying the length of a line by the number of lines. This is what gives the impression of random access to the data in the file.

Where am I?

To determine where we are in a file we can use the tell() method of a file. For example if I open a file and read three lines, I can then ask the file how far into the file I am.

Let's look at an example, first I will create a file with 5 lines of text all the same length (the equal length business isn't strictly necessary but it does make life easier!). Then I'll read three lines back and ask where we are. I'll then go back to the beginning, read one line then jump to the third line and print it, jumping over the second line. Like this:

# create 5 lines of 20 chars (+ \n)
testfile = open('testfile.txt','w')
for i in range(5):
    testfile.write(str(i) * 20 + '\n')
testfile.close()

# read 3 lines and ask where we are
testfile = open('testfile.txt','r')
for line in range(3):
    print( testfile.readline().strip() )
position = testfile.tell()
print( "At position: ", position, "bytes" )

# go back to the beginning
testfile.seek(0)
print( testfile.readline().strip() ) # repeat first line
lineLength = testfile.tell()
testfile.seek(2*lineLength)  # go to end of line 2
print( testfile.readline().strip() ) # the 3rd line
testfile.close()

Note the use of the seek() function to move the cursor. The default operation is to move it to the byte number specified, as shown here, but extra arguments can be provided that change the indexing method used. Also note that the value printed by the first tell() depends on the length of a newline on your platform, on my Windows 10 PC it printed 66 indicating that the newline sequence is 2 bytes long. But since this is a platform specific value and I want to make my code portable I've used tell() again, after reading one line, to work out how long each line really is. These kind of "cunning ploys" are often necessary when dealing with platform specific issues!