TensorFlow

Article Info

Contributed by
4 authors

Last updated on
2019-11-04 06:46:58

Improve this article

TensorLayer
TensorBoard
Tensor Processing Unit
Deep Learning
Reinforcement Learning
Keras

Article Versions

8 2019-11-04 06:46:58
1729,1728 8,1729

By arvindpdmn

Spelling correction.
7 2019-11-04 06:32:40
1728,1653 7,1728

By arvindpdmn

Comparing TF2.0 vs TF1.x. Updated Keras related milestone. New TF2 sample code.
6 2019-10-01 05:47:16
1653,923 6,1653

By arvindpdmn

TF 2.0 added. New image added.
5 2018-08-29 14:27:24
923,148 5,923

By gurumoorthyP

Major updates based on latest developments. Old content improved and reorganized.
4 2017-07-09 14:36:53
148,145 4,148

By arvindpdmn

Published article. Formatting changes. Changed the tags. Spelling correction.

Chat Room

Submitting ...

You are editing an existing chat message.
2017-07-19 03:42:29
Discussion

By arvindpdmn

One question can talk about data flow graph along with an image. This can also mention TensorBoard (which can be included in See Also section). More references and citations needed.

TensorFlow logo. Source: TensorFlow GitHub 2018.

TensorFlow is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) that flow between them. This dataflow paradigm enables parallelism, distributed execution, optimal compilation and portability.

The typical use of TensorFlow is for Machine Learning (ML), particularly Deep Learning (DL) that uses large scale multi-layered neural networks. More specifically, it's best for classification, perception, understanding, discovery, prediction and creation.

TensorFlow was originally developed by researchers and engineers working on the Google Brain team within Google's Machine Intelligence Research organization for ML/DL research. The system is general enough to be applicable in a wide variety of other domains as well.

Discussion

For which use cases is TensorFlow best suited?
TensorFlow can be used in any domain where ML/DL can be employed. It can also be used in other forms of AI, including reinforcement learning and logistic regression. On mobile devices, applications include speech recognition, image recognition, object localization, gesture recognition, optical character recognition, translation, text classification, voice synthesis, and more.
Some of the areas are:
- Voice/Speech Recognition: For voice-based interfaces as popularized by Apple Siri, Amazon Alexa or Microsoft Cortana. For sentiment analysis in CRM. For flaw detection (noise analysis) in industrial systems.
- Text-Based Applications: For sentimental analysis (CRM, Social Media), threat detection (Social Media, Government) and fraud detection (Insurance, Finance). For machine translation such as with Google Translate. For text summarization using sequence-to-sequence learning. For language detection. For automated email replies such as with Google SmartReply.
- Image Recognition: For face recognition, image search, machine vision and photo clustering. For object classification and identification within larger images. For cancer detection in medical applications.
- Time-Series Analysis: For forecasting. For customer recommendations. For risk detection, predictive analytics and resource planning.
- Video Detection: For motion detection in gaming and security systems. For large-scale video understanding.
Could you name some applications where TensorFlow is being used?
A TensorFlow introduction that mentions real-world use cases. Source: Google Developers YouTube 2017.
TensorFlow is being used by Google in following areas:
- RankBrain: Google search engine.
- SmartReply: Deep LSTM model to automatically generate email responses.
- Massively Multitask Networks for Drug Discovery: A deep neural network model for identifying promising drug candidates.
- On-Device Computer Vision for OCR - On-device computer vision model to do optical character recognition to enable real-time translation.
- Retinal imaging - Early detection of diabetic retinopathy using deep neural network of 26 layers.
- SyntaxNet - Built for Natural Language Understanding (NLU), this is based on TensorFlow and open sourced by Google in 2016.
Outside Google, we mention some known real-world examples. Mozilla uses TensorFlow for speech recognition. UK supermarket Ocado uses it for route planning for its robots, demand forecasting, and product recommendations. A Japanese farmer has used it to classify cucumbers based on shape, length and level of distortion. As an experiment, Intel used TensorFlow on traffic videos for pedestrian detection.
Further examples were noted at the TensorFlow Developer Summit, 2018.
Which platforms and languages support TensorFlow?
TensorFlow is available on 64-bit Linux, macOS, Windows and also on the mobile computing platforms like Android and iOS. Google has announced a software stack specifically for Android development called TensorFlow Lite.
TensorFlow has official APIs available in the following languages: Python, JavaScript, C++, Java, Go, Swift. Python API is recommended. Bindings in other languages are available from community: C#, Haskell, Julia, Ruby, Rust, Scala. There's also C++ API reference for TensorFlow Serving. R's tensorflow package provides access to the complete TensorFlow API from within R.
Nvidia's TensorRT, a Programmable Inference Accelerator, allows you to optimize your models for inference by lowering precision and thereby reducing latency.
How is TensorFlow different from other ML/DL platforms?
GitHub stars of some ML frameworks shows TensorFlow's rising popularity. Source: Maciej 2016.
TensorFlow is relatively painless to setup. With its growing community adoption, it offers a healthy ecosystem of updates, tutorials and example code. It can run on a variety of hardware. It's cross platform. It has APIs or bindings in many popular programming languages. It supports GPU acceleration. Through TensorBoard, you get an intuitive view of your computation pipeline. Keras, a DL library, can run on TensorFlow. However, it's been criticized for being more complex and slower than alternative frameworks.
Created in 2007, Theano is one of the first DL frameworks but it's been perceived as too low-level. Support for Theano is also ending. Written in Lua, Torch is meant for GPUs. It's Python port released by Facebook, called PyTorch, is popular for analyzing unstructured data. It's developer friendly and memory efficient. Caffe2 does well for modeling convolutional neural networks. Apache MXNet, along with its simplified DL interface called Gluon, is supported by Amazon and Microsoft. Microsoft also has Microsoft Cognitive Toolkit (CNTK) that can handle large datasets. For Java and Scala programmers, there's Deeplearning4j.
Which are the tools closely related to TensorFlow?
TensorFlow pipeline and key components. Source: Synced 2019.
The following are closely associated with or variants of TensorFlow:
- TensorFlow Lite: Enables low-latency inferences on mobile and embedded devices.
- TensorFlow Mobile: To use TensorFlow from within iOS or Android mobile apps, where TensorFlow Lite cannot be used.
- TensorFlow Serving: A high performance, open source serving system for machine learning models, designed for production environments and optimized for TensorFlow.
- TensorLayer: Provides popular DL and RL modules that can be easily customized and assembled for tackling real-world machine learning problems.
- TensorFlow Hub: A library for the publication, discovery, and consumption of reusable parts of machine learning models.
- TensorFlow Model Analysis: A library for evaluating TensorFlow models.
- TensorFlow Debugger: Allows us to view the internal structure and states of running TensorFlow graphs during training and inference.
- TensorFlow Playground: A browser-based interface for beginners to tinker with neural networks. Written in TypeScript and D3.js. Doesn't actually use TensorFlow.
- TensorFlow.js: Build and train models entirely in the browser or Node.js runtime.
- TensorBoard: A suite of visualization tools that helps to understand, debug, and optimize TensorFlow programs.
- TensorFlow Transform: A library for preprocessing data with TensorFlow.
What's the architecture of TensorFlow?
The architecture of TensorFlow. Source: TensorFlow Team 2017b.
TensorFlow can be deployed across platforms, details of which are abstracted away from higher layers. The core itself is implemented in C++ and exposes its features via APIs in many languages, with Python being the most recommended.
Above these language APIs, is the Layers API that offers commonly used layers in deep learning models. To read data, Datasets API is the recommended way and it creates input pipelines. With Estimators, we can create custom models or bring in models pre-made for common ML tasks.
XLA (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that optimizes TensorFlow computations. If offers improvements in speed, memory usage, and portability on server and mobile platforms.
Could you explain how TensorFlow's data graph works?
A simple TensorFlow graph. Source: Khan 2018.
TensorFlow uses a dataflow graph, which is a common programming model for parallel computing. Graph nodes represent operations and edges represent data consumed or produced by the nodes. Edges are called tensors that carry data. In the example figure, we show five graph nodes: a and b are placeholders to accept inputs; c, d and e are simple arithmetic operations.
In TensorFlow 1.x, when a graph is created, tensors don't contain the results of operations. The graph is evaluated through sessions, which encapsulate the TensorFlow runtime. However, with eager execution, operations are evaluated immediately instead of building a graph for later execution. This is useful for debugging and iterating quickly on small models or data.
For ingesting data into the graph, placeholders can be used for the simplest cases but otherwise, datasets should be preferred. To train models, layers are used to modify values in the graph.
To simplify usage, high-level API called estimators should be used. They encapsulate training, evaluation, prediction and export for serving. Estimators themselves are built on layers and build the graph for you.
How is TensorFlow 2.0 different from TensorFlow 1.x?
Comparing TF1.x versus TF2.0. Source: Lazy Programmer 2019.
It makes sense to write any new code in TensorFlow 2.0. Existing 1.x code can be migrated to 2.0. The recommended path is to move to TensorFlow 1.14 and then to 2.0. Compatibility module tf.compat should help.
Here are the key changes in TensorFlow 2.0:
- API Cleanup: Many APIs are removed or moved. For example, absl-py package replaces tf.app, tf.flags, and tf.logging. Main namespace tf.* is cleaned up by moving some items into subpackages such as tf.math. Examples of new modules are tf.summary, tf.keras.metrics, and tf.keras.optimizers.
- Eager Execution: Like Python, eager execution is the default behaviour. Code execute in order, making tf.control_dependencies() redundant.
- No More Globals: We need to keep track of variables. Untracked tf.Variable will get garbage collected.
- Functions, Not Sessions: Functions are more familiar to developers. Although session.run() is gone, for efficiency and JIT compilation, tf.function() decorator can be used. This automatically invokes AutoGraph to convert Python constructs into TensorFlow graph equivalents. Functions can be shared and reused.

Milestones

2011

Five-layer deep neural network partitioned across four machines in DisBelief. Source: Dean et al. 2012, fig. 1.

Google Brain invents DistBelief, a framework to train large models for machine learning. DistBelief can make use of computing clusters of thousands of machines for accelerated training. The framework manages details of parallelism (multithreading, message passing), synchronization and communication. Compared to MapReduce, DistBelief is better at deep network training. Compared to GraphLab, DistBelief is better at structured graphs.

Nov
2015

Under Apache 2.0 licensing, Google open sources TensorFlow, which is Google Brain's second-generation machine learning system. While other open source ML frameworks exist (Caffe, Theano, Torch), Google's competence in ML is supposedly 5-7 years ahead of the rest. However, Google doesn't open source algorithms that run on TensorFlow, not its advanced hardware infrastructure.

Apr
2016

Version 0.8 of TensorFlow is released. It comes with distributed training support. Powered by gRPC, models can be trained on hundreds of machines in parallel. For example, Inception image classification network was trained using 100 GPUs with an overall speedup of 56x compared to a single GPU. More generally, the system can map the dataflow graph onto heterogeneous devices (multi-core CPUs, general-purpose GPUs, mobile processors) in the available processes.

May
2016

Google announced that it's been using Tensor Processing Unit (TPU), a custom ASIC built specifically for machine learning and tailored for TensorFlow.

Jun
2016

TensorFlow v0.9 is released with support for iOS and Raspberry Pi. Android support has been around from the beginning.

Feb
2017

Version 1.0 of TensorFlow is released. The API is in Python but there's also experimental APIs in Java and Go.

Nov
2017

Google releases a preview of TensorFlow Lite for mobile and embedded devices. This enables low-latency inferences for on-device ML models. In future, this should be preferred over TensorFlow Mobile. With TensorFlow 1.4, we can build models using high-level Keras API. Keras, which was previously in tf.contrib.keras, is now the core package tf.keras.

Sep
2019

TensorFlow 2.0 is released following an alpha release in June. It improves workflows for both production and experimentation. It promises better performance on GPU acceleration.

Sample Code

# Source: https://www.tensorflow.org/tutorials/quickstart/beginner
# Accessed: 2019-11-04
 
from __future__ import absolute_import, division, print_function, unicode_literals
 
import tensorflow as tf
 
mnist = tf.keras.datasets.mnist
 
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
 
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])
 
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
 
model.fit(x_train, y_train, epochs=5)
 
model.evaluate(x_test,  y_test, verbose=2)

# Source: https://www.tensorflow.org/get_started/get_started
# Accessed: 2017-07-09
 
import numpy as np
import TensorFlow as tf
 
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Model input and output
x = tf.placeholder(tf.float32)
linear_model = W * x + b
y = tf.placeholder(tf.float32)
# loss
loss = tf.reduce_sum(tf.square(linear_model - y)) # sum of the squares
# optimizer
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)
# training data
x_train = [1,2,3,4]
y_train = [0,-1,-2,-3]
# training loop
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init) # reset values to wrong
for i in range(1000):
  sess.run(train, {x:x_train, y:y_train})
 
# evaluate training accuracy
curr_W, curr_b, curr_loss = sess.run([W, b, loss], {x:x_train, y:y_train})
print("W: %s b: %s loss: %s"%(curr_W, curr_b, curr_loss))

# Use of R's tensorflow package to access TensorFlow API
# Source: https://tensorflow.rstudio.com/tensorflow/articles/using_tensorflow_api.html
# Accessed: 2018-08-28
 
library(tensorflow)
 
# Create 100 phony x, y data points, y = x * 0.1 + 0.3
x_data <- runif(100, min=0, max=1)
y_data <- x_data * 0.1 + 0.3
 
# Try to find values for W and b that compute y_data = W * x_data + b
# (We know that W should be 0.1 and b 0.3, but TensorFlow will
# figure that out for us.)
W <- tf$Variable(tf$random_uniform(shape(1L), -1.0, 1.0))
b <- tf$Variable(tf$zeros(shape(1L)))
y <- W * x_data + b
 
# Minimize the mean squared errors.
loss <- tf$reduce_mean((y - y_data) ^ 2)
optimizer <- tf$train$GradientDescentOptimizer(0.5)
train <- optimizer$minimize(loss)
 
# Launch the graph and initialize the variables.
sess = tf$Session()
sess$run(tf$global_variables_initializer())
 
# Fit the line (Learns best fit is W: 0.1, b: 0.3)
for (step in 1:201) {
  sess$run(train)
  if (step %% 20 == 0)
    cat(step, "-", sess$run(W), sess$run(b), "\n")
}

// Source: https://js.tensorflow.org/
// Accessed: 2018-08-29
// The following runs in Node.js environment
const tf = require('@tensorflow/tfjs');
 
// Load the binding:
require('@tensorflow/tfjs-node');  // Use '@tensorflow/tfjs-node-gpu' if running with GPU.
 
// Train a simple model:
const model = tf.sequential();
model.add(tf.layers.dense({units: 100, activation: 'relu', inputShape: [10]}));
model.add(tf.layers.dense({units: 1, activation: 'linear'}));
model.compile({optimizer: 'sgd', loss: 'meanSquaredError'});
 
const xs = tf.randomNormal([100, 10]);
const ys = tf.randomNormal([100, 1]);
 
model.fit(xs, ys, {
  epochs: 100,
  callbacks: {
    onEpochEnd: async (epoch, log) => {
      console.log(`Epoch ${epoch}: loss = ${log.loss}`);
    }
  }
});