• TensorFlow logo. Source: TensorFlow GitHub 2018.
    image
  • Five-layer deep neural network partitioned across four machines in DisBelief. Source: Dean et al. 2012, fig. 1.
    image
  • GitHub stars of some ML frameworks shows TensorFlow's rising popularity. Source: Maciej 2016.
    image
  • The architecture of TensorFlow. Source: TensorFlow Team 2017b.
    image
  • A simple TensorFlow graph. Source: Khan 2018.
    image

TensorFlow

Summary

image
TensorFlow logo. Source: TensorFlow GitHub 2018.

TensorFlow is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) that flow between them. This dataflow paradigm enables parallelism, distributed execution, optimal compilation and portability.

The typical use of TensorFlow is for Machine Learning (ML), particularly Deep Learning (DL) that uses large scale multi-layered neural networks. More specifically, it's best for classification, perception, understanding, discovery, prediction and creation.

TensorFlow was originally developed by researchers and engineers working on the Google Brain team within Google's Machine Intelligence Research organization for ML/DL research. The system is general enough to be applicable in a wide variety of other domains as well.

Milestones

2011
image

Google Brain invents DistBelief, a framework to train large models for machine learning. DistBelief can make use of computing clusters of thousands of machines for accelerated training. The framework manages details of parallelism (multithreading, message passing), synchronization and communication. Compared to MapReduce, DistBelief is better at deep network training. Compared to GraphLab, DistBelief is better at structured graphs.

Nov
2015

Under Apache 2.0 licensing, Google open sources TensorFlow, which is Google Brain's second-generation machine learning system. While other open source ML frameworks exist (Caffe, Theano, Torch), Google's competence in ML is supposedly 5-7 years ahead of the rest. However, Google doesn't open source algorithms that run on TensorFlow, not its advanced hardware infrastructure.

Apr
2016

Version 0.8 of TensorFlow is released. It comes with distributed training support. Powered by gRPC, models can be trained on hundreds of machines in parallel. For example, Inception image classification network was trained using 100 GPUs with an overall speedup of 56x compared to a single GPU. More generally, the system can map the dataflow graph onto heterogeneous devices (multi-core CPUs, general-purpose GPUs, mobile processors) in the available processes.

May
2016

Google announced that it's been using Tensor Processing Unit (TPU), a custom ASIC built specifically for machine learning and tailored for TensorFlow.

Jun
2016

TensorFlow v0.9 is released with support for iOS and Raspberry Pi. Android support has been around from the beginning.

Feb
2017

Version 1.0 of TensorFlow is released. The API is in Python but there's also experimental APIs in Java and Go.

Nov
2017

Google releases a preview of TensorFlow Lite for mobile and embedded devices. This enables low-latency inferences for on-device ML models. In future, this should be preferred over TensorFlow Mobile.

Aug
2018

TensorFlow version 1.10 is released.

Discussion

  • For which use cases is TensorFlow best suited?

    TensorFlow can be used in any domain where ML/DL can be employed. It can also be used in other forms of AI, including reinforcement learning and logistic regression. On mobile devices, applications include speech recognition, image recognition, object localization, gesture recognition, optical character recognition, translation, text classification, voice synthesis, and more.

    Some of the areas are:

    • Voice/Speech Recognition: For voice-based interfaces as popularized by Apple Siri, Amazon Alexa or Microsoft Cortana. For sentiment analysis in CRM. For flaw detection (noise analysis) in industrial systems.
    • Text-Based Applications: For sentimental analysis (CRM, Social Media), threat detection (Social Media, Government) and fraud detection (Insurance, Finance). For machine translation such as with Google Translate. For text summarization using sequence-to-sequence learning. For language detection. For automated email replies such as with Google SmartReply.
    • Image Recognition: For face recognition, image search, machine vision and photo clustering. For object classification and identification within larger images. For cancer detection in medical applications.
    • Time-Series Analysis: For forecasting. For customer recommendations. For risk detection, predictive analytics and resource planning.
    • Video Detection: For motion detection in gaming and security systems. For large-scale video understanding.
  • Could you name some applications where TensorFlow is being used?
    A TensorFlow introduction that mentions real-world use cases. Source: Google Developers YouTube 2017.

    TensorFlow is being used by Google in following areas:

    • RankBrain: Google search engine.
    • SmartReply: Deep LSTM model to automatically generate email responses.
    • Massively Multitask Networks for Drug Discovery: A deep neural network model for identifying promising drug candidates.
    • On-Device Computer Vision for OCR - On-device computer vision model to do optical character recognition to enable real-time translation.
    • Retinal imaging - Early detection of diabetic retinopathy using deep neural network of 26 layers.
    • SyntaxNet - Built for Natural Language Understanding (NLU), this is based on TensorFlow and open sourced by Google in 2016.

    Outside Google, we mention some known real-world examples. Mozilla uses TensorFlow for speech recognition. UK supermarket Ocado uses it for route planning for its robots, demand forecasting, and product recommendations. A Japanese farmer has used it to classify cucumbers based on shape, length and level of distortion. As an experiment, Intel used TensorFlow on traffic videos for pedestrian detection.

    Further examples were noted at the TensorFlow Developer Summit, 2018.

  • Which platforms and languages support TensorFlow?

    TensorFlow is available on 64-bit Linux, macOS, Windows and also on the mobile computing platforms like Android and iOS. Google has announced a software stack specifically for Android development called TensorFlow Lite.

    TensorFlow has official APIs available in the following languages: Python, JavaScript, C++, Java, Go, Swift. Python API is recommended. Bindings in other languages are available from community: C#, Haskell, Julia, Ruby, Rust, Scala. There's also C++ API reference for TensorFlow Serving. R's tensorflow package provides access to the complete TensorFlow API from within R.

    Nvidia's TensorRT, a Programmable Inference Accelerator, allows you to optimize your models for inference by lowering precision and thereby reducing latency.

  • How is TensorFlow different from other ML/DL platforms?
    image
    GitHub stars of some ML frameworks shows TensorFlow's rising popularity. Source: Maciej 2016.

    TensorFlow is relatively painless to setup. With its growing community adoption, it offers a healthy ecosystem of updates, tutorials and example code. It can run on a variety of hardware. It's cross platform. It has APIs or bindings in many popular programming languages. It supports GPU acceleration. Through TensorBoard, you get an intuitive view of your computation pipeline. Keras, a DL library, can run on TensorFlow. However, it's been criticized for being more complex and slower than alternative frameworks.

    Created in 2007, Theano is one of the first DL frameworks but it's been perceived as too low-level. Support for Theano is also ending. Written in Lua, Torch is meant for GPUs. It's Python port released by Facebook, called PyTorch, is popular for analyzing unstructured data. It's developer friendly and memory efficient. Caffe2 does well for modeling convolutional neural networks. Apache MXNet, along with its simplified DL interface called Gluon, is supported by Amazon and Microsoft. Microsoft also has Microsoft Cognitive Toolkit (CNTK) that can handle large datasets. For Java and Scala programmers, there's Deeplearning4j.

  • Which are the tools closely related to TensorFlow?

    The following are closely associated with or variants of TensorFlow:

    • TensorFlow Lite: Enables low-latency inferences on mobile and embedded devices.
    • TensorFlow Mobile: To use TensorFlow from within iOS or Android mobile apps, where TensorFlow Lite cannot be used.
    • TensorFlow Serving: A high performance, open source serving system for machine learning models, designed for production environments and optimized for TensorFlow.
    • TensorLayer: Provides popular DL and RL modules that can be easily customized and assembled for tackling real-world machine learning problems.
    • TensorFlow Hub: A library for the publication, discovery, and consumption of reusable parts of machine learning models.
    • TensorFlow Model Analysis: A library for evaluating TensorFlow models.
    • TensorFlow Debugger: Allows us to view the internal structure and states of running TensorFlow graphs during training and inference.
    • TensorFlow Playground: A browser-based interface for beginners to tinker with neural networks. Written in TypeScript and D3.js. Doesn't actually use TensorFlow.
    • TensorFlow.js: Build and train models entirely in the browser or Node.js runtime.
    • TensorBoard: A suite of visualization tools that helps to understand, debug, and optimize TensorFlow programs.
    • TensorFlow Transform: A library for preprocessing data with TensorFlow.
  • What's the architecture of TensorFlow?
    image
    The architecture of TensorFlow. Source: TensorFlow Team 2017b.

    TensorFlow can be deployed across platforms, details of which are abstracted away from higher layers. The core itself is implemented in C++ and exposes its features via APIs in many languages, with Python being the most recommended.

    Above these language APIs, is the Layers API that offers commonly used layers in deep learning models. To read data, Datasets API is the recommended way and it creates input pipelines. With Estimators, we can create custom models or bring in models pre-made for common ML tasks.

    XLA (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that optimizes TensorFlow computations. If offers improvements in speed, memory usage, and portability on server and mobile platforms.

  • Could you explain how TensorFlow's data graph works?
    image
    A simple TensorFlow graph. Source: Khan 2018.

    TensorFlow uses a dataflow graph, which is a common programming model for parallel computing. Graph nodes represent operations and edges represent data consumed or produced by the nodes. Edges are called tensors that carry data. In the example figure, we show five graph nodes: a and b are placeholders to accept inputs; c, d and e are simple arithmetic operations.

    When a graph is created, tensors don't contain the results of operations. The graph is evaluated through sessions, which encapsulate the TensorFlow runtime. However, TensorFlow has the concept of eager execution as well. With eager execution, operations are evaluated immediately instead of building a graph for later execution. This is useful for debugging and iterating quickly on small models or data.

    For ingesting data into the graph, placeholders can be used for the simplest cases but otherwise, datasets should be preferred. To train models, layers are used to modify values in the graph.

    To simplify usage, high-level API called estimators should be used. They encapsulate training, evaluation, prediction and export for serving. Estimators themselves are built on layers and build the graph for you.

Sample Code

  • # Source: https://www.tensorflow.org/get_started/get_started; Accessed 2017-07-09.
     
    import numpy as np
    import TensorFlow as tf
     
    # Model parameters
    W = tf.Variable([.3], dtype=tf.float32)
    b = tf.Variable([-.3], dtype=tf.float32)
    # Model input and output
    x = tf.placeholder(tf.float32)
    linear_model = W * x + b
    y = tf.placeholder(tf.float32)
    # loss
    loss = tf.reduce_sum(tf.square(linear_model - y)) # sum of the squares
    # optimizer
    optimizer = tf.train.GradientDescentOptimizer(0.01)
    train = optimizer.minimize(loss)
    # training data
    x_train = [1,2,3,4]
    y_train = [0,-1,-2,-3]
    # training loop
    init = tf.global_variables_initializer()
    sess = tf.Session()
    sess.run(init) # reset values to wrong
    for i in range(1000):
      sess.run(train, {x:x_train, y:y_train})
     
    # evaluate training accuracy
    curr_W, curr_b, curr_loss = sess.run([W, b, loss], {x:x_train, y:y_train})
    print("W: %s b: %s loss: %s"%(curr_W, curr_b, curr_loss))

References

  1. Carey, Scott. 2017. "What is TensorFlow? How are businesses using it?" Computer World UK, November 17. Accessed 2018-08-29.
  2. Dean, Jeffrey, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang and Andrew Y. Ng. 2012. "Large Scale Distributed Deep Networks." NIPS 2012. Accessed 2018-08-28.
  3. Exastax. 2017. "Top Five Use Cases of TensorFlow." Exastax, February 03. Accessed 2018-08-29.
  4. Exastax. 2018. "A Comparison of Deep Learning Frameworks." Exastax, February 26. Accessed 2018-08-28.
  5. Fiedel, Noah. 2016. "Running your models in production with TensorFlow Serving." Google AI Blog, February 16. Accessed 2018-08-28.
  6. Google Developers YouTube. 2017. "TensorFlow: Machine Learning for Everyone." Google Developers YouTube, February 15. Accessed 2018-08-29.
  7. Gupta, Sandeep. 2018. "Highlights from the TensorFlow Developer Summit, 2018." TensorFlow, Medium, March 30. Accessed 2018-08-28.
  8. Jouppi, Norm. 2016. "Google supercharges machine learning tasks with TPU custom chip." Google Cloud Platform Blog. May 18. Accessed 2018-08-28.
  9. Khan, Tiamur. 2018. "Making chatbots more human using machine learning." mc.ai, April 30. Accessed 2018-08-29.
  10. Maciej. 2016. "Machine Learning Frameworks Comparison." Paperspace Blog, November 11. Accessed 2018-08-29.
  11. Metz, Cade. 2015. "Google Just Open Sourced TensorFlow, Its Artificial Intelligence Engine." Wired, November 09. Accessed 2018-08-28.
  12. Moroney, Laurence. 2018. "Using TensorFlow Lite on Android." Medium, March 31. Accessed 2018-08-28.
  13. Murray, Derek. 2016. "Announcing TensorFlow 0.8 – now with distributed computing support!" Google AI Blog, April 13. Accessed 2018-08-28.
  14. Nvidia. 2018. "NVIDIA TensorRT: Programmable Inference Accelerator." Accessed 2018-08-28.
  15. Peng, Lily. 2017. "Case Study: TensorFlow in Medicine - Retinal Imaging." TensorFlow Dev Summit 2017, Google Developers, YouTube, February 15. Accessed 2018-08-29.
  16. Petrov, Slav. 2016. "Announcing SyntaxNet: The World’s Most Accurate Parser Goes Open Source." Google AI Blog, May 12. Accessed 2018-08-28.
  17. Piqueras, Gema Parreno. 2016. "TensorFlow Introduction." Medium, April 13. Accessed 2018-08-29.
  18. Pookalangara, Ajit Kumar, Rajeswari Ponnuru, and Ravi Keron Nidamarty. 2018. "Pedestrian Detection Using TensorFlow* on Intel® Architecture." Intel AI Academy, January 30. Updated 2018-04-27. Accessed 2018-08-29.
  19. Sandjideh, Amy McDonald. 2017. "Announcing TensorFlow 1.0." Google Developers, February 15. Accessed 2018-08-28.
  20. Sato, Kaz. 2016. "How a Japanese cucumber farmer is using deep learning and TensorFlow." Blog, Google Cloud, August 31. Accessed 2018-08-29.
  21. TensorFlow. 2018a. "Hub." TensorFlow. Accessed 2018-08-28.
  22. TensorFlow. 2018b. "TensorBoard: Visualizing Learning." TensorFlow, August 08. Accessed 2018-08-28.
  23. TensorFlow. 2018c. "Introduction to TensorFlow Mobile." TensorFlow, July 31. Accessed 2018-08-28.
  24. TensorFlow. 2018d. "TensorFlow Debugger." TensorFlow, August 08. Accessed 2018-08-28.
  25. TensorFlow. 2018e. "XLA Overview." TensorFlow, August 08. Accessed 2018-08-28.
  26. TensorFlow. 2018f. "Graphs and Sessions." TensorFlow, August 08. Accessed 2018-08-28.
  27. TensorFlow. 2018g. "Estimators." TensorFlow, August 25. Accessed 2018-08-28.
  28. TensorFlow. 2018h. "Eager Execution." TensorFlow, August 08. Accessed 2018-08-28.
  29. TensorFlow. 2018i. "Low Level APIs: Introduction." TensorFlow, August 08. Accessed 2018-08-28.
  30. TensorFlow. 2018j. "TensorFlow In Use." TensorFlow, August 08. Accessed 2018-08-28.
  31. TensorFlow Docs. 2018. "API Documentation." August 25. Accessed 2018-08-29.
  32. TensorFlow GitHub. 2018. "TensorFlow." TensorFlow Github. July 17. Accessed 2017-07-07.
  33. TensorFlow Team. 2017a. "Announcing TensorFlow Lite." Google Developers, November 14. Accessed 2018-08-28.
  34. TensorFlow Team. 2017b. "Introduction to TensorFlow Datasets and Estimators." Google Developers, September 12. Accessed 2018-08-28.
  35. TensorFlow for R. 2018. "R Interface to TensorFlow." Accessed 2018-08-29.
  36. TensorLayer Docs. 2018. "Welcome to TensorLayer." Version 1.9.1, Revision 9528da50, August. Accessed 2018-08-28.
  37. Unruh, Amy. 2017. "What is the TensorFlow machine intelligence platform?" Opensource.com, November 09. Accessed 2018-08-28.
  38. Vincent, James. 2017. Google’s new machine learning framework is going to put more AI on your phone." The Verge, May 17. Accessed 2018-08-29.
  39. Warden, Pete. 2016. "TensorFlow v0.9 now available with improved mobile support." Google Developers, June 27. Accessed 2018-08-28.
  40. Wikipedia. 2018. "TensorFlow." Wikipedia, August 11. Accessed 2018-08-28.

Milestones

2011
image

Google Brain invents DistBelief, a framework to train large models for machine learning. DistBelief can make use of computing clusters of thousands of machines for accelerated training. The framework manages details of parallelism (multithreading, message passing), synchronization and communication. Compared to MapReduce, DistBelief is better at deep network training. Compared to GraphLab, DistBelief is better at structured graphs.

Nov
2015

Under Apache 2.0 licensing, Google open sources TensorFlow, which is Google Brain's second-generation machine learning system. While other open source ML frameworks exist (Caffe, Theano, Torch), Google's competence in ML is supposedly 5-7 years ahead of the rest. However, Google doesn't open source algorithms that run on TensorFlow, not its advanced hardware infrastructure.

Apr
2016

Version 0.8 of TensorFlow is released. It comes with distributed training support. Powered by gRPC, models can be trained on hundreds of machines in parallel. For example, Inception image classification network was trained using 100 GPUs with an overall speedup of 56x compared to a single GPU. More generally, the system can map the dataflow graph onto heterogeneous devices (multi-core CPUs, general-purpose GPUs, mobile processors) in the available processes.

May
2016

Google announced that it's been using Tensor Processing Unit (TPU), a custom ASIC built specifically for machine learning and tailored for TensorFlow.

Jun
2016

TensorFlow v0.9 is released with support for iOS and Raspberry Pi. Android support has been around from the beginning.

Feb
2017

Version 1.0 of TensorFlow is released. The API is in Python but there's also experimental APIs in Java and Go.

Nov
2017

Google releases a preview of TensorFlow Lite for mobile and embedded devices. This enables low-latency inferences for on-device ML models. In future, this should be preferred over TensorFlow Mobile.

Aug
2018

TensorFlow version 1.10 is released.

Tags

See Also

Further Reading

  1. TensorFlow. 2018i. "Low Level APIs: Introduction." TensorFlow, August 08. Accessed 2018-08-28.
  2. TensorFlow Team. 2017b. "Introduction to TensorFlow Datasets and Estimators." Google Developers, September 12. Accessed 2018-08-28.
  3. Metz, Cade. 2015. "Google Just Open Sourced TensorFlow, Its Artificial Intelligence Engine." Wired, November 09. Accessed 2018-08-28.
  4. Exastax. 2017. "Top Five Use Cases of TensorFlow." Exastax, February 03. Accessed 2018-08-29.
  5. Spann, Tim. 2017. "TensorFlow for Real-World Applications." DZone, October 17. Accessed 2018-08-29.

Top Contributors

Last update: 2018-08-29 14:27:24 by gurumoorthyP
Creation: 2017-07-09 11:25:55 by ashish2702

Article Stats

1600
Words
1
Chats
4
Authors
5
Edits
3
Likes
1277
Hits

Cite As

Devopedia. 2018. "TensorFlow." Version 5, August 29. Accessed 2018-09-25. https://devopedia.org/tensorflow
BETA V0.17