When we encounter a new case (a new input/output relationship), we first check if it is familiar to us.
If it is, there is no spontaneous learning effort involved. Later the new case is merged into our knowledge system.
If it is not, we might cram this unfamiliar case first. The cramming results in a strict rule with respect to this unfamiliar case. Then we will soften the strictness of the new case and do our best to merge the new case into our knowledge system.
面對的問題如果是曾經有過的資料，則會選擇相關的資訊回應。
面對的問題如果是未曾有過的資料，則會選擇增加一個新回應。
ASLFN (3層的類神經網路 輸入->隱藏->輸出層)
The Adaptive Single-hidden Layer Feed-forward Neural Networks (ASLFN, i.e., the amount of adopted hidden nodes is variable) (references: Tsaih 1993; Tsaih 1998; Tsaih and Cheng 2009; Huang, Yu, Tsaih and Huang Tsaih 2014; Kuo, Lin, and Hsu 2018)
2-class categorization problem and binary inputs {−1, 1}^𝑚
Activation Function (激活函數)
tanh, the hyperbolic tangent activation function, is used in all hidden nodes.
The Activation Value of Hidden Nodes (隱藏層的激活值)
激活函數不能亂給，會有梯度爆炸的情形發生。
$𝐱^𝑐≡(𝑥_1^𝑐,𝑥_2^𝑐, …,𝑥_𝑚^𝑐 )T：𝑡ℎ𝑒 input vector of the 𝑐^th case$
$𝑤_{𝑖0}^{𝐻}：the threshold value of the 𝑖^th hidden node$
$𝑤_𝑖𝑗^𝐻：the weight between the 𝑖^th hidden node and the 𝑗^th input node$
The Activation Value of the Output Node (輸出層的激活數值)
Parameters and Indexes (參數)
N denotes the number of all reference observations
m denotes the number of input nodes
p denotes the number of adopted hidden nodes; p equals 1 at the beginning and is adaptive
$𝑦^𝑐$ denotes the desired output of the $𝑐^th$ case, with 1.0 and -1.0 being the desired outputs of classes 1 and 2
n denotes the $𝑛^th$ stage of handling n reference observations ${(𝐱^1, 𝑦^1), (𝐱^2, 𝑦^2), …, (𝐱^𝑛, 𝑦^𝑛)}$, and $𝐈(𝑛)$ is the set of indices of these observations.
At the $𝑛^th$ stage, the loss function $𝐸𝑛 (𝐰)≡\frac{∑{𝑐=1}^{𝑛}(𝑓(𝐱^𝑐,𝐰)−𝑦^𝑐 )^2}{𝑛}+10^{-3}‖𝐰‖^2$
$𝐈(𝑛)≡𝐈_1 (𝑛)∪𝐈_2 (𝑛)$, where$𝐈_1(𝑛)$ and $𝐈_2(𝑛)$ are the set of indices of n given cases in classes 1 and 2
At the $𝑛^th$ stage with the reference observations ${(𝐱^𝑐, 𝑦^𝑐): 𝑐∈𝐈(𝑛)}$, we look for an acceptable SLFN, in which the condition L regarding ${𝑓(𝐱^𝑐,𝐰), ∀ 𝑐∈𝐈(𝑛)}$ is satisfied.
The condition 𝐿 regarding ${𝑓(𝐱^𝑐,𝐰), ∀ 𝑐∈𝐈(𝑛)}$
The Learning Goal (學習目標)
At the $𝑛^{th}$ stage, through minimizing the loss function
$𝐸𝑛 (𝐰)≡\frac{∑{𝑐=1}^{𝑛}(𝑓(𝐱^𝑐,𝐰)−𝑦^𝑐 )^2}{𝑛}+10^{-3} (∑_{𝑖=0}^{𝑝}{(w_𝑖^o)^2} + ∑_{𝑖=1}^{𝑝}∑_{𝑗=0}^{𝑚}(𝑤_{𝑖𝑗}^{𝐻})^2)$,the learning goal is to seek $𝐰$ where $𝑓(𝐱^𝑐,𝐰)>𝜈,∀ 𝑐∈𝐈_1 (𝑛)$ and $𝑓(𝐱^𝑐,𝐰)≤−𝜈,∀ 𝑐∈𝐈_2(𝑛)$, with $1>𝜈>0$. An alternative goal of learning is to seek 𝐰 that satisfies the condition 𝐿 regarding ${𝑓(𝐱^𝑐,𝐰), ∀ 𝑐∈𝐈(𝑛)}$
When $𝛼>𝛽$ is satisfied, $𝑓(𝐱^𝑐,𝐰)≥𝑣 ,∀ 𝑐∈𝐈_1(𝑛)$and $𝑓(𝐱^𝑐,𝐰)≤−𝑣,∀ 𝑐∈𝐈_2(𝑛)$ can be achieved by directly adjusting $𝐰^𝑜$ according to the following:
The Proposed CSM Learning Algorithm
Step 1: Initialize a SLFN with one hidden node with a randomized w. Obtain the first reference observation $(𝐱^1, 𝑦^1)$. Set $n = 2$.
Step 2: If $n > N$, STOP.
Step 3: Present the n reference observations ${(𝐱^𝑐, 𝑦^𝑐): c ∈ I(n)}$.
Step 4: If the condition L regarding ${𝑓(𝐱^𝑐,𝐰), ∀ 𝑐∈𝐈(𝑛)}$ is satisfied, go to Step 7.1.
Step 5: Save $w$.
Step 6: Apply the weight-tuning mechanism to $min_{𝐰}{𝐸(𝐰)}$ to adjust $w$ until one of the following two cases occurs:
a. If the condition L regarding {𝑓(𝐱^𝑐,𝐰), ∀ 𝑐∈𝐈(𝑛)} is satisfied, go to step 7.1.
b. If an unacceptable result is obtained, then
Restore w
Let $p + 1 → p$ and add to the existing SLFN the new $𝑝^{th}$ hidden node with the following $𝐰_𝑝^𝐻$ and $𝐰_𝑝^𝑜$:
Step 7.1: Apply the weight-tuning mechanism one hundred times to minimizing $𝐸_𝑛(𝐰)$ to adjust $𝐰$, while keeping the condition L regarding ${𝑓(𝐱^𝑐,𝐰), ∀ 𝑐∈𝐈(𝑛)}$ satisfied.
Step 7.3: If $- \theta >max_{1<k<p}{g_{k}^{‘}}$, where $\theta$ is a given constant, go to Step 2.
Step 7.4: If $max_{1<k<p}^{𝑔𝑘^′} > 0$, prune the $𝑖^{th}$ hidden node, in which 𝑖 is the first index of $arg max{1<k<p}𝑔_{𝑘}^{′}$, $p-1->p$, $𝐰_𝑖^′->w$, and go to Step 7.1.
Flowchart of the proposed algorithm (流程圖)
Explanation of the Proposed CSM Learning Algorithm (解釋CSM學習演算法)
Step 6 conducts the cramming mechanism.
The Appendix shows the properness of the cramming mechanism.
All hidden nodes use the same activation function, but some of them have the heterogeneity due to their large associated weights.
The total amount of used hidden nodes will be large if new hidden nodes are added frequently.
Step 7.1 and Step 7.5.c are designed to soften the heterogeneity.
At Step 6, Step 7.1 and Step 7.5.c, the weight-tuning mechanism is applied to minimizing $𝐸_𝑛(𝐰)$ to adjust weights, while there is the regularization term in $𝐸_𝑛(𝐰)$.
The total amount of used hidden nodes will be large if new hidden nodes are added frequently.
Step 7.2 to Step 7.5 are designed to merge the unfamiliar case into the knowledge system via reducing the total amount of used hidden nodes.
The pruning issue file shows the logic of the merging mechanism.
The irrelevant hidden node
The irrelevance examination mechanism
No under-fitting
The Regularization term in $𝐸_𝑛(𝐰)$
Regarding the overfitting issue, the regularization term in the loss function $𝐸_𝑛(𝐰)$ may prevent the model from doing too well on training data.
]]><h2 id="類神經網路-歷史"><a href="#類神經網路-歷史" class="headerlink" title="類神經網路 歷史"></a>類神經網路 歷史</h2><p>人工智慧最早出現於1950年代。人工智慧的目標是希望能讓電腦像人一班思考與學習。被視為人工智EM演算法https://caocharles.github.io/EM%E6%BC%94%E7%AE%97%E6%B3%95/2018-08-28T01:31:35.000Z2026-07-19T13:58:59.632Z參考網址 Maximum Likelihood from Incomplete Data via the EM Algorithm
$\Longleftrightarrow$ Find the expectation of the complete-data loglikelihood with respect to the missing data y given the observed data x and the current parameter estimates $\theta^{(t)}$.
197 animals are distributed multinomially into four categories with cell-probabilities$(\frac{1}{2}+ \frac{\theta}{4}, \frac{(1-\theta)}{4}, \frac{(1-\theta)}{4}, \frac{\theta}{4})$, where $\theta \in (0,1)$is unknown
TensorFlow’s capability to execute the code on different devices such as CPUs and GPUs is a consequence of it’s specific structure:
TensorFlow defines computations as Graphs, and these are made with operations (also know as “ops”). So, when we work with TensorFlow, it is the same as defining a series of operations in a Graph.
To execute these operations as computations, we must launch the Graph into a Session. The session translates and passes the operations represented into the graphs to the device you want to execute them on, be it a GPU or CPU.
For example, the image below represents a graph in TensorFlow. W, x and b are tensors over the edges of this graph. MatMul is an operation over the tensors W and x, after that Add is called and add the result of the previous operator with b. The resultant tensors of each operation cross the next one until the end where it’s possible to get the wanted result.
Importing TensorFlow
To use TensorFlow, we need to import the library. We imported it and optionally gave it the name “tf”, so the modules can be accessed by tf.module-name:
1
import tensorflow as tf
—————–
Building a Graph
As we said before, TensorFlow works as a graph computational model. Let’s create our first graph.
To create our first graph we will utilize source operations, which do not need any information input. These source operations or source ops will pass their information to other operations which will execute computations.
To create two source operations which will output numbers we will define two constants:
1 2
a = tf.constant([2]) b = tf.constant([3])
After that, let’s make an operation over these variables. The function tf.add() adds two elements (you could also use c = a + b).
1 2
c = tf.add(a,b) #c = a + b is also a way to define the sum of the terms
Then TensorFlow needs to initialize a session to run our code. Sessions are, in a way, a context for creating a graph inside TensorFlow. Let’s define our session:
1
session = tf.Session()
Let’s run the session to get the result from the previous defined ‘c’ operation:
1 2
result = session.run(c) print(result)
Close the session to release resources:
1
session.close()
To avoid having to close sessions every time, we can define them in a with block, so after running the with block the session will close automatically:
1 2 3
with tf.Session() as session: result = session.run(c) print(result)
Even this silly example of adding 2 constants to reach a simple result defines the basis of TensorFlow. Define your edge (In this case our constants), include nodes (operations, like tf.add), and start a session to build a graph.
What is the meaning of Tensor?
In TensorFlow all data is passed between operations in a computation graph, and these are passed in the form of Tensors, hence the name of TensorFlow.
The word tensor from new latin means “that which stretches”. It is a mathematical object that is named tensor because an early application of tensors was the study of materials stretching under tension. The contemporary meaning of tensors can be taken as multidimensional arrays.
That’s great, but… what are these multidimensional arrays?
Going back a little bit to physics to understand the concept of dimensions:
The zero dimension can be seen as a point, a single object or a single item.
The first dimension can be seen as a line, a one-dimensional array can be seen as numbers along this line, or as points along the line. One dimension can contain infinite zero dimension/points elements.
The second dimension can be seen as a surface, a two-dimensional array can be seen as an infinite series of lines along an infinite line.
The third dimension can be seen as volume, a three-dimensional array can be seen as an infinite series of surfaces along an infinite line.
The Fourth dimension can be seen as the hyperspace or spacetime, a volume varying through time, or an infinite series of volumes along an infinite line. And so forth on…
Now that you understand these data structures, I encourage you to play with them using some previous functions to see how they will behave, according to their structure types:
with tf.Session() as session: result = session.run(first_operation) print ("Defined using tensorflow function :") print(result) result = session.run(second_operation) print ("Defined using normal expressions :") print(result)
With the regular symbol definition and also the TensorFlow function we were able to get an element-wise multiplication, also known as Hadamard product.
But what if we want the regular matrix product?
We then need to use another TensorFlow function called tf.matmul():
with tf.Session() as session: result = session.run(first_operation) print ("Defined using tensorflow function :") print(result)
We could also define this multiplication ourselves, but there is a function that already does that, so no need to reinvent the wheel!
Why Tensors?
The Tensor structure helps us by giving the freedom to shape the dataset the way we want.
And it is particularly helpful when dealing with images, due to the nature of how information in images are encoded,
Thinking about images, its easy to understand that it has a height and width, so it would make sense to represent the information contained in it with a two dimensional strucutre (a matrix)… until you remember that images have colors, and to add information about the colors, we need another dimension, and thats when Tensors become particulary helpful.
Images are encoded into color channels, the image data is represented into each color intensity in a color channel at a given point, the most common one being RGB, which means Red, Blue and Green. The information contained into an image is the intensity of each channel color into the width and height of the image, just like this:
So the intensity of the red channel at each point with width and height can be represented into a matrix, the same goes for the blue and green channels, so we end up having three matrices, and when these are combined they form a tensor.
Variables
Now that we are more familiar with the structure of data, we will take a look at how TensorFlow handles variables.
To define variables we use the command tf.variable(). To be able to use variables in a computation graph it is necessary to initialize them before running the graph in a session. This is done by running tf.global_variables_initializer().
To update the value of a variable, we simply run an assign operation that assigns a value to the variable:
1
state = tf.Variable(0)
Let’s first create a simple counter, a variable that increases one unit at a time:
Variables must be initialized by running an initialization operation after having launched the graph. We first have to add the initialization operation to the graph:
1
init_op = tf.global_variables_initializer()
We then start a session to run the graph, first initialize the variables, then print the initial value of the state variable, and then run the operation of updating the state variable and printing the result after each update:
1 2 3 4 5 6
with tf.Session() as session: session.run(init_op) print(session.run(state)) for _ inrange(3): session.run(update) print(session.run(state))
Placeholders
Now we know how to manipulate variables inside TensorFlow, but what about feeding data outside of a TensorFlow model?
If you want to feed data to a TensorFlow model from outside a model, you will need to use placeholders.
So what are these placeholders and what do they do?
Placeholders can be seen as “holes” in your model, “holes” which you will pass the data to, you can create them using tf.placeholder(datatype), where datatype specifies the type of data (integers, floating points, strings, booleans) along with its precision (8, 16, 32, 64) bits.
The definition of each data type with the respective python sintax is defined as:
Data type
Python type
Description
DT_FLOAT
tf.float32
32 bits floating point.
DT_DOUBLE
tf.float64
64 bits floating point.
DT_INT8
tf.int8
8 bits signed integer.
DT_INT16
tf.int16
16 bits signed integer.
DT_INT32
tf.int32
32 bits signed integer.
DT_INT64
tf.int64
64 bits signed integer.
DT_UINT8
tf.uint8
8 bits unsigned integer.
DT_STRING
tf.string
Variable length byte arrays. Each element of a Tensor is a byte array.
DT_BOOL
tf.bool
Boolean.
DT_COMPLEX64
tf.complex64
Complex number made of two 32 bits floating points: real and imaginary parts.
DT_COMPLEX128
tf.complex128
Complex number made of two 64 bits floating points: real and imaginary parts.
Now we need to define and run the session, but since we created a “hole” in the model to pass the data, when we initialize the session we are obligated to pass an argument with the data, otherwise we would get an error.
To pass the data to the model we call the session with an extra argument feed_dict in which we should pass a dictionary with each placeholder name folowed by its respective data, just like this:
1 2 3
with tf.Session() as sess: result = sess.run(b,feed_dict={a:3.5}) print (result)
Since data in TensorFlow is passed in form of multidimensional arrays we can pass any kind of tensor through the placeholders to get the answer to the simple multiplication operation:
with tf.Session() as sess: result = sess.run(b,feed_dict=dictionary) print (result)
Operations
Operations are nodes that represent the mathematical operations over the tensors on a graph. These operations can be any kind of functions, like add and subtract tensor or maybe an activation function.
tf.matmul, tf.add, tf.nn.sigmoid are some of the operations in TensorFlow. These are like functions in python but operate directly over tensors and each one does a specific thing.
a = tf.constant([5]) b = tf.constant([2]) c = tf.add(a,b) d = tf.subtract(a,b)
with tf.Session() as session: result = session.run(c) print ('c =: %s' % result) result = session.run(d) print ('d =: %s' % result)
tf.nn.sigmoid is an activiation function, it’s a little more complicated, but this function helps learning models to evaluate what kind of information is good or not.
Want to learn more?
Running deep learning programs usually needs a high performance platform. PowerAI speeds up deep learning and AI. Built on IBM’s Power Systems, PowerAI is a scalable software platform that accelerates deep learning and AI with blazing performance for individual users or enterprises. The PowerAI platform supports popular machine learning libraries and dependencies including Tensorflow, Caffe, Torch, and Theano. You can download a free version of PowerAI.
Also, you can use Data Science Experience to run these notebooks faster with bigger datasets. Data Science Experience is IBM’s leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, DSX enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of DSX users today with a free account at Data Science ExperienceThis is the end of this lesson. Hopefully, now you have a deeper and intuitive understanding regarding the LSTM model. Thank you for reading this notebook, and good luck on your studies.
]]><p><a href="https://caocharles.github.io/">連結測試</a></p>
<p>打打看文字。</p>
<p>12345</p>
<figure class="highlight plaintext"><table><tr><td class=Hello Worldhttps://caocharles.github.io/hello-world/2018-08-26T23:58:15.056Z2026-07-19T13:58:59.632ZWelcome to Hexo! This is your very first post. Check documentation for more info. If you get any problems when using Hexo, you can find the answer in troubleshooting or you can ask me on GitHub.
