What exactly is TensorFlow?

All About Google’s Latest Open-Source AI

TensorFlow is an open-source second generation of the Google Brain system. Verion 1.0.0 was first released in February 11, 2017. TensorFlow is capable of running on multiple CPU and GPU platforms.  And with optional CUDA extensions, TensorFLow can be used on a wide-variety of conventional  64-bit operating systems including Windows, macOS,  and Linux. TensorFlow can also be used on mobile devices both with iOS and Andoid.   TensorFlow is used to train neural networks for tasks such as detecting and deciphering patterns, correlations, and analogous. It’s similar to the way humans learn and reason, and It has been used for research at Google. Tensorflow is currently being integrated into many different cloud-based applications like the Kaldi speech recognition platform.
Google’s machine intelligence framework is the talk of the town right now. TensorFlow is available on the Raspberry Pi, so it became very easy and affordable to work with. In a small amount of time, I was able to create a neural network that has the capability to count in binary. So, let me pass on exactly what I have learned so far. If I have done my job correctly, it should be easy enough for anyone that would like to attempt this.  

So what is TensorFlow exactly?

  To quote the developers, TensorFlow is an “open-source library for numbered computation by way of data flow graphs”. What are “data glow graphs”? that’s the compelling part of this article. However, let’s start with the basics of a neural networks.   A basic neural network has several input units. It also has “ghost” or hidden units. This is because, from a typical users view, they are literally hidden from plain sight. And there are also output units which display the results. Not often mentioned are the bias units, which are there to control the values generated from the ghost and output units. Connecting all of these parts together are a “load” of weights, which are just numerical identifiers, both of which are connected with two units.   Now we have to instill intelligence into the neural network by assigning values to the weights. Basically, that’s the biggest part of training a neural network. We seek suitable values for those weights. Once they have been trained, we can go about setting the input units with the binary digits 0, 0, and 0 accordingly, TensorFlow will operate with everything in between, and then the output units will contain the digits 0, 0 and 1 accordingly. If you missed that part, it was “aware” that the logical process after 000 was 001. For 001, it should then give 010, and eventually to 111, at that point it will give 000. When these weights are correct, it will have learned how to count.   One of the more important steps in “operating” the network is to multiply the value given to each weight by the value of its current input unit, and it will then store that result in the associated “ghost” unit.   It is possible to redraw units and weights into arrays, defined as “lists” in the Python Language. Using math terminology, They’re matrices. We have them redrawn in our diagram. With simple matrix multiplication we can multiply the input matrix with our weight matrix resulting in a five element hidden matric/list/array.
In TensorFLow, lists are called tensors. Our matrix multiplication step is referred to as an operation, or “op”, this is a term that you must get used to using if you plan on studying any part of the TensorFlow documentation. If we go any farther, the entirety of our neural network is just a coagulation of tensors and ops that operate inside of one another. All together this is called a “graph” The tensors are displayed as lines, those marks on the lines are the tensors dimensions. The tensors are connected by the ops, even though most of the things you can see can be clicked for a closer look. We have already done that for layer1 in our second snapshot.
At the lowest point is displayed X, this is the variable assigned just for the purpose of a placeholder op that will allow us to provide sample inputs to the tensor. The horizontal line rising and then rearing to the left is our input tensor. Keep following that line and what you will find is the MatMul op, this is where the matrix multiplication is handled using the input tensor and the alternative tensor which leads to MatMul op. This tensor represents the weights. To this point I have described quite a bit about tensors and ops. Providing you with more understanding about what I mean by TensorFlow. “open-source library for numbered computation by way of data flow graphs” But what purpose exactly do we have in creating these graphs exactly? The API that is currently being used is designed for Python, this if you don’t already know is an interpreted language. Neural networks are extremely resource intensive. A Large network could have up to millions upon millions of weights. If we were to interpret every single step it would take a very large amount of time. What we do instead is to create a graph that is made up of tensors and ops. Then we describe the layout  of the neural network, with all of the mathematical operations and initial values using variables. Only after we have created the graph does it then become what is referred to in TensorFlow as a session. This is called deferred execution. Our session runs the graph using only extremely efficient code. A large number of the operations, like matrix multiplication, can be done on supported GPU’s “Graphics-Processing-Unit” and the session can handle it for you. Also, TensorFlow is designed with the capability for distributing the load across several machines or – supported GPU’s. only access to the complete graph allows us to do that. Below is the code for our example binary counter neural network. I will include a link to the GitHub page where you can find the source code. IMPORTANT there is additional code with the purpose of saving important information using TensorBoard. Let’s get started and create a graph of tensors and ops; import tensorflow as tf sess = tf.InteractiveSession() NUM_INPUTS = 3 NUM_HIDDEN = 5 NUM_OUTPUTS = 3 We begin by importing the tensorflow module, open a “session” for our uses later in,  and to make our code to be more discernable, let’s create several variable containing the exact number of units in our small network. x = tf.placeholder(tf.float32, shape=[None, NUM_INPUTS], name=’x’) y_ = tf.placeholder(tf.float32, shape=[None, NUM_OUTPUTS], name=’y_’) Placeholders are necessary for input and output units. A placeholder is simply a temporary TensorFlow op to be used for our designated purposes later. X and y_ will now be tensors in our new graph both will have a placeholder op connected with it. So, if you are wondering why we call the shapes as [None, NUM_INPUTS] and [None, NUM_OUTPUTS], as two dimensional lists, and why None for our first dimension? In our code above, it may appear that we’ll give it one input at a time then “train” it to create its output. It will be far more efficient though, if and when we provide it with multiple forms of input/output pairs in each group. There is no way to know just how many will be in that group until we actually assign one at a later date. In fact, we are already using the very same graph for training, testing, and for actual usage. The group size really won’t ever be exactly the same. That is why we will be using the Python placeholder object None for the scape of our first dimension at this moment. W_fc1 = tf.truncated_normal([NUM_INPUTS, NUM_HIDDEN], mean=0.5, stddev=0.707) W_fc1 = tf.Variable(W_fc1, name=’W_fc1′) b_fc1 = tf.truncated_normal([NUM_HIDDEN], mean=0.5, stddev=0.707) b_fc1 = tf.Variable(b_fc1, name=’b_fc1′) h_fc1 = tf.nn.relu(tf.matmul(x, W_fc1) + b_fc1) That is preceeded by generating the first layer of our neural network graph: the weights W_fc1, the biases b_fc1, and the hidden units h_fc1. The “fc” is a convention meaning “fully connected”, since the weights connect every input unit to every ghost unit. tf.truncated_normal results in an immediate number of ops and tensors which will be late assigned normalized, random numbers to all the weights. All of our variable ops will be granted a given value to begin initialization with. These are purely random numbers They will keep their given data across numerous runs. They will come in handy for saving our neural network to a file, something you’ll really want to do once it has been trained.  
W_fc2 = tf.truncated_normal([NUM_HIDDEN, NUM_OUTPUTS], mean=0.5, stddev=0.707) W_fc2 = tf.Variable(W_fc2, name=’W_fc2′) b_fc2 = tf.truncated_normal([NUM_OUTPUTS], mean=0.5, stddev=0.707) b_fc2 = tf.Variable(b_fc2, name=’b_fc2′) y = tf.matmul(h_fc1, W_fc2) + b_fc2
what exactly is tensorflow?
Our weights and biases on layer two will be set as seen above ^ to be the exact same for layer one. However, the output layer will be different. Back to matrix multiplication, multiply our weights and – Yes the ghost units. Then simply add the bias weights. And that leaves our activation function for the preceeding code; results = tf.sigmoid(y, name=’results’) cross_entropy = tf.reduce_mean( tf.nn.sigmoid_cross_entropy_with_logits(logits=y, labels=y_))   Sigmoid is yet another activation function, its purpose – to provide non-linearity. And this is why I used it here because it results in values between 0 and 1. This was perfect for the counter example. Sigmoid is also king when it comes to dealing with larger values. In this case, it will handle the value 111. All of the outputs can results in large values. With image classification we’d want something just a bit different, what would be best could be one output unit able to fire a large value. Just as an example, let’s use a giraffe. This will represent a large image value, if this was out-target we’d want to use something like softmax.   With a closer look, it would appear like there might be some slight duplication. It might appear to the reader like we have mistakenly inserted sigmoid twice in our code. This is intentional with the purpose of creating two different paralleled outputs. Our cross entropy tensor will be used during the training process of our neural network. Our results tensor is simply going to be used after we have ran our trained neural network. But simply for example and fun in the case!   Hey, this might not be considered the “best way” but let’s take a shot!   train_step = tf.train.RMSPropOptimizer(0.25, momentum=0.5).minimize(cross_entropy)   So, that brings us to the last part of our graph which doesn’t leave us with much but the training. Now, Here we go! It will begin with the op or ops, and they will adjust out weights based on all of the available training data. Lets remember though that this is very important. The only purpose is to create a graph. Any actual training will occur later when we have actually run the graph.   With several optimizers to choose from, I chose tf.train.RMSPropOptimizer, The reason for this is because it, -like sigmoid, is great when handling large output values. If we were classifying things like when doing image classification,  tf.train.GradientDescentOptimizer  it is probably going to be of more use.   Now we get to my favorite part! Training and using our new Binary Counter.  Now that the graph has been created, we have to begin the training. Once that is done we can then use it. inputvals = [[0, 0, 0], [0, 0, 1], [0, 1, 0], [0, 1, 1], [1, 0, 0], [1, 0, 1], [1, 1, 0], [1, 1, 1]] targetvals = [[0, 0, 1], [0, 1, 0], [0, 1, 1], [1, 0, 0], [1, 0, 1], [1, 1, 0], [1, 1, 1], [0, 0, 0]]   To start, let me show you my training data: inputvals and targetvals they contain the inputs, for each of their corresponding targetvals target value. With inputvals[0] theres, [0, 0, 0], and then the expected output will be targetvals[0], which will be [0, 0, 1], and so on, so on. if do_training == 1: sess.run(tf.global_variables_initializer()) for i in range(10001): if i%100 == 0: train_error = cross_entropy.eval(feed_dict={x: inputvals, y_:targetvals}) print(“step %d, training error %g”%(i, train_error)) if train_error < 0.0005: break sess.run(train_step, feed_dict={x: inputvals, y_: targetvals}) if save_trained == 1: print(“Saving neural network to %s.*”%(save_file)) saver = tf.train.Saver() saver.save(sess, save_file) do_training and save_trained have to ability to be hardcoded, and manipulated for each specific use but they can also be manually set using arguments in the command line.   First we have to allow the variable ops initialize their tensors.   For all the way up to 10001 times the graph will run from the bottom to the train_step tensor. This is the last thing we added to the graphs. Send inputvals and targetvals to the train_step’s op or ops, those of which  we will add using RMSPropOptimizer. This will allow all of the weights to be adjusted for given inputs this will give us the results something close to the corresponding target outputs. If an error is given within one of the target outputs and the main outputs gets small enough quicker, we will lose our loop.   When you have thousand of inputs/outputs subsets can be given at a specific time, that group we spoke of earlier in the article. Good news is that in this article we only have eight! So, we will provide them each time.   If we desired so, our new network can be saved to a file -also mentioned previously. And once it has been thoroughly trained adn prepared. It will not have to be trained again.   else: # if training has already been completed we can load it from a file. print(“Loading neural network from %s”%(save_file)) saver = tf.train.Saver() saver.restore(sess, save_file) # Note: restore allows us to load the variables and data for the trained network   If we have already trained an existing network, we have no need to do it here. We can load the saved network into this code. This file will only contain the values for the tensors that have existing variable ops. This doesn’t mean that it contains the structure of the graphs- and it won’t. So if we are already running a trained graph, the necessary code to create the graph is still needed. If we were interested in saving graphs instead, there are tools available for such. MetaGraphs is a tool designed specifically for that purpose. However in this example we will not be doing that. print(‘\nCounting starting with: 0 0 0’) res = sess.run(results, feed_dict={x: [[0, 0, 0]]}) print(‘%g %g %g’%(res[0][0], res[0][1], res[0][2])) for i in range(8): res = sess.run(results, feed_dict={x: res}) print(‘%g %g %g’%(res[0][0], res[0][1], res[0][2]))   In both cases, we are going to try it out. You will notice that we are going from the bottom of the graph up to the results tensor like we talked about earlier. This duplicate output was created specifically for us to use when working with an already trained network.   We through 000, and cross our fingers that our result will be something remotely close to 001. And pass the return. Back to it we go and run again. After it has completed an operation nine-times enough for the result to go from 000 to 111 and once again return to 000.   Now let’s take a look at the results and see what we get. These will be after a fully successful training and counting exercise. You will notice based on the results that it trained after only 200 steps through our loop. Every now and again it might be capable of all 10001 steps without reducing the training errors. But if you’ve trained successfully and saved it, it really won’t matter.   All of the code used in this example is available on the GitHub page. For learning purposes I would personally recommend starting from scratch. But keep in mind that available on the TensorFlow website are a great many tutorials these are extremely valuable.   Personally my next accomplishment will be using TensorFlow to control a form a hardware. You have heard of a gentleman by the name of Lukas Biewald? He is using TensorFlow and built a robot that could recognize objects around his workshop. It’s truly amazing what you can do with this tool! Thanks for reading this post by the crew at Webology