I'm using a custom tf.Estimator object to train a neural network. The problem is in size of the events file after training - it is unreasonably large. I've already solved the problem with saving part of a dataset as constant by using tf.Dataset.from_generator(). However, the size is still quite large and while starting tensorboard I'm getting the message
W0225 10:38:07.443567 140693578311424 tf_logging.py:120] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
So, I suppose, that I'm creating and saving many different graphs in this event file. Is it possible to turn off this saving or how to save an only first copy?
For know, I found only the way to delete all the default logs by deleting the events filts with
list(map(os.remove, glob.glob(os.path.join(runtime_params['model_dir'], 'events.out.tfevents*'))))
However, it is bad solution for me, as I would prefer to keep the summaries and, ideally, one copy of the graph.
From documentation, I can see that
Estimators automatically write the following to disk:
- checkpoints, which are versions of the model created during training.
- event files, which contain information that TensorBoard uses to create visualizations. Is it a way to turn off writitng of the events file?
from How to create only one copy of graph in tensorboard events file with custom tf.Estimator?
No comments:
Post a Comment