tf.io.gfile.GFile

File I/O wrappers without thread locking.

tf.io.gfile.GFile(
    name, mode='r'
)

Used in the notebooks

Used in the tutorials

The main roles of the tf.io.gfile module are:

  1. To provide an API that is close to Python's file I/O objects, and
  2. To provide an implementation based on TensorFlow's C++ FileSystem API.

The C++ FileSystem API supports multiple file system implementations, including local files, Google Cloud Storage (using a gs:// prefix, and HDFS (using an hdfs:// prefix). TensorFlow exports these as tf.io.gfile, so that you can use these implementations for saving and loading checkpoints, writing to TensorBoard logs, and accessing training data (among other uses). However, if all your files are local, you can use the regular Python file API without any problem.

Once you obtain a GFile object, you can use it in most ways as you would any Python's file object:

with open("/tmp/x", "w") as f:
  f.write("asdf")
4
with tf.io.gfile.GFile("/tmp/x") as f:
  f.read()
'asdf'

The difference is that you can specify URI schemes to use other filesystems (e.g., gs:// for GCS, s3:// for S3, etc.), if they are supported. Using file:// as an example, we have:

with tf.io.gfile.GFile("file:///tmp/x", "w") as f:
  f.write("qwert")
  f.write("asdf")
tf.io.gfile.GFile("file:///tmp/x").read()
'qwertasdf'

You can also read all lines of a file directly:

with tf.io.gfile.GFile("file:///tmp/x", "w") as f:
  f.write("asdf\n")
  f.write("qwer\n")
tf.io.gfile.GFile("/tmp/x").readlines()
['asdf\n', 'qwer\n']

You can iterate over the lines:

with tf.io.gfile.GFile("file:///tmp/x", "w") as f:
  f.write("asdf\n")
  f.write("qwer\n")
for line in tf.io.gfile.GFile("/tmp/x"):
  print(line[:-1]) # removes the end of line character
asdf
qwer

Random access read is possible if the underlying filesystem supports it:

with open("/tmp/x", "w") as f:
  f.write("asdfqwer")
f = tf.io.gfile.GFile("/tmp/x")
f.read(3)
'asd'
f.seek(4)
f.tell()
4
f.read(3)
'qwe'
f.tell()
7
f.close()

mode Returns the mode in which the file was opened.
name Returns the file name.

Methods

close

View source

close()

Closes the file.

Should be called for the WritableFile to be flushed.

In general, if you use the context manager pattern, you don't need to call this directly.

with tf.io.gfile.GFile("/tmp/x", "w") as f:
  f.write("asdf\n")
  f.write("qwer\n")
# implicit f.close() at the end of the block

For cloud filesystems, forgetting to call close() might result in data loss as last write might not have been replicated.

flush

View source

flush()

Flushes the Writable file.

This only ensures that the data has made its way out of the process without any guarantees on whether it's written to disk. This means that the data would survive an application crash but not necessarily an OS crash.

next

View source

next()

read

View source

read(
    n=-1
)

Returns the contents of a file as a string.

Starts reading from current position in file.

Args
n Read n bytes if n != -1. If n = -1, reads to end of file.

Returns
n bytes of the file (or whole file) in bytes mode or n bytes of the string if in string (regular) mode.

readline

View source

readline()

Reads the next line, keeping \n. At EOF, returns ''.

readlines

View source

readlines()

Returns all lines from the file in a list.

seek

View source

seek(
    offset=None, whence=0, position=None
)

Seeks to the offset in the file. (deprecated arguments)

Args
offset The byte count relative to the whence argument.
whence Valid values for whence are: 0: start of the file (default) 1: relative to the current position of the file 2: relative to the end of file. offset is usually negative.

seekable

View source

seekable()

Returns True as FileIO supports random access ops of seek()/tell()

size

View source

size()

Returns the size of the file.

tell

View source

tell()

Returns the current position in the file.

write

View source

write(
    file_content
)

Writes file_content to the file. Appends to the end of the file.

__enter__

View source

__enter__()

Make usable with "with" statement.

__exit__

View source

__exit__(
    unused_type, unused_value, unused_traceback
)

Make usable with "with" statement.

__iter__

View source

__iter__()