Skip to content
Snippets Groups Projects
Commit d8244b74 authored by schmittu's avatar schmittu :beer:
Browse files

minor stuff

parent ae1d7eb5
No related branches found
No related tags found
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
02_classification.ipynb
+ 22
− 4
View file @ d8244b74
This diff is collapsed.
create_datasets.py.ipynb
+ 375
− 1
View file @ d8244b74
This diff is collapsed.
neural_nets_intro.ipynb
+ 9
− 19
View file @ d8244b74
%% Cell type:markdown id: tags:
# Introduction to Neural Networks
## TO DO: Almost all the figues and schematics will be replaced or improved slowly
<img src="./images/neuralnets/Colored_neural_network.svg"/>
source: https://en.wikipedia.org/wiki/Artificial_neural_network
%% Cell type:markdown id: tags:
## History of Neural networks
**TODO: Make it more complete and format properly**
1943 - Threshold Logic
1940s - Hebbian Learning
1958 - Perceptron
1975 - Backpropagation
1980s - Neocognitron
1982: Hopfield Network
1986: Convolutional Neural Networks
1997: Long-short term memory (LSTM) model
2014: GRU, GAN ?
%% Cell type:markdown id: tags:
## Building blocks
### Perceptron
Smallest unit of a neural network is a **perceptron** like node.
**What is a Perceptron?**
It is a simple function which has multiple inputs and a single output.
Step 1: Weighted sum of the inputs is calculated
\begin{equation*}
weighted\_sum = \sum_{k=1}^{num\_inputs} w_{i} x_{i}
\end{equation*}
Step 2: The following activation function is applied
$$
f(weighted\_sum) = \left\{
\begin{array}{ll}
0 & \quad weighted\_sum < threshold \\
1 & \quad weighted\_sum \geq threshold
\end{array}
\right.
$$
You can see that this is also a linear classifier as we introduced in script 02.
%% Cell type:code id: tags:
``` python
%matplotlib inline
%config IPCompleter.greedy=True
```
%% Cell type:code id: tags:
``` python
import numpy as np
def perceptron(X, w, threshold=1):
# This function computes sum(w_i*x_i) and
# applies a perceptron activation
linear_sum = np.dot(X,w)
output=0
if linear_sum >= threshold:
output = 1
# print("The perceptron has peaked")
return output
X = [1,0]
w = [1,1]
perceptron(X,w)
```
%% Output
1
%% Cell type:markdown id: tags:
#### Boolean AND
| x$_1$ | x$_2$ | output |
| --- | --- | --- |
| 0 | 0 | 0 |
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 1 | 1 |
%% Cell type:code id: tags:
``` python
# Calculating Boolean AND using a perceptron
import matplotlib.pyplot as plt
threshold = 1.5
w=[1,1]
X=[[0,0],[1,0],[0,1],[1,1]]
for i in X:
print("Perceptron output for x1, x2 = " + str(i)[1:-1] + " is " + str(perceptron(i,w,threshold)))
```
%% Output
Perceptron output for x1, x2 = 0, 0 is 0
Perceptron output for x1, x2 = 1, 0 is 0
Perceptron output for x1, x2 = 0, 1 is 0
Perceptron output for x1, x2 = 1, 1 is 1
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
In this simple case we can rewrite our equation to $x_2 = ...... $ which describes a line in 2D:
%% Cell type:code id: tags:
``` python
# Plotting the decision boundary
plt.xlim(-1,2)
plt.ylim(-1,2)
for i in X:
plt.plot(i,"o",color="b");
# Plotting the decision boundary
# that is a line given by w_1*x_1+w_2*x_2-threshold=0
plt.plot(np.arange(-3,4), 1.5-np.arange(-3,4), "--", color="black");
```
%% Output
%% Cell type:markdown id: tags:
**Exercise :Can you compute a Boolean "OR" using a perceptron?**
Hint: copy the code from the "AND" example and edit the weights and/or threshold
%% Cell type:markdown id: tags:
#### Boolean OR
| x$_1$ | x$_2$ | output |
| --- | --- | --- |
| 0 | 0 | 0 |
| 1 | 0 | 1 |
| 0 | 1 | 1 |
| 1 | 1 | 1 |
%% Cell type:code id: tags:
``` python
# Calculating Boolean OR using a perceptron
# Edit the code below
```
%% Cell type:code id: tags:
``` python
# Solution
# Calculating Boolean OR using a perceptron
import matplotlib.pyplot as plt
threshold=0.6
w=[1,1]
X=[[0,0],[1,0],[0,1],[1,1]]
for i in X:
print("Perceptron output for x1, x2 = " + str(i)[1:-1] + " is " + str(perceptron(i,w,threshold)))
# Plotting the decision boundary
plt.xlim(-1,2)
plt.ylim(-1,2)
for i in X:
plt.plot(i,"o",color="b");
# Plotting the decision boundary
# that is a line given by w_1*x_1+w_2*x_2-threshold=0
plt.plot(np.arange(-3,4), threshold-np.arange(-3,4), "--", color="black");
```
%% Output
Perceptron output for x1, x2 = 0, 0 is 0
Perceptron output for x1, x2 = 1, 0 is 1
Perceptron output for x1, x2 = 0, 1 is 1
Perceptron output for x1, x2 = 1, 1 is 1
%% Cell type:markdown id: tags:
**Optional exercise: Create a NAND gate with perceptrons**
%% Cell type:code id: tags:
``` python
# Calculating Boolean NAND using a perceptron
```
%% Cell type:markdown id: tags:
In fact a single perceptron can compute "AND", "OR" and "NOT" boolean functions.
However, it cannot compute some other boolean functions such as "XOR"
WHAT CAN WE DO?
Hint: What is the significance of the NAND gate we created above
We said a single perceptron can't compute these functions. We didn't say that about **multiple Perceptrons**
%% Cell type:markdown id: tags:
**XOR function**
**TO DO: INSERT IMAGE HERE!!!!!!!!!!!!!!**
%% Cell type:markdown id: tags:
### Activation Functions
Some of the commonly used activation functions
%% Cell type:code id: tags:
``` python
import matplotlib.pyplot as plt
import numpy as np
plt.figure(figsize=(10, 4), constrained_layout=False)
plt.subplot(1, 3, 1)
#fig, ax = plt.subplots(1,3)
#ax=ax.flatten()
pts=np.arange(-20,20, 0.1)
# Sigmoid
plt.plot(pts, 1/(1+np.exp(-pts))) ;
plt.subplot(1, 3, 2)
# tanh
plt.plot(pts, np.tanh(pts*np.pi)) ;
# Rectified linear unit (ReLu)
plt.subplot(1, 3, 3)
pts_relu=[max(0,i) for i in pts];
plt.plot(pts, pts_relu) ;
```
%% Output
%% Cell type:markdown id: tags:
Suggestion Uwe:
1. more layers might improve power of single perctptron.
2. regrettably math show that just "stacking" perceptrons only adds little improvements
3. way around: look at nature how neuron works and introduce non linear activation functions.
4. theoretical background: universal approximation theorem.
### Multi-layer preceptron neural network
Universal function theorem
#### Gradient based learning
#### back propogation
#### loss function
#### optimization
epochs
### Google Playground
UWE: move up before discussing gradient stuff etc
https://playground.tensorflow.org/
<img src="./images/neuralnets/google_playground.png"/>
%% Cell type:markdown id: tags:
# Introduction to Keras
%% Cell type:markdown id: tags:
What is **Keras**?
* It is a high level API to create and work with neural networks
* Supports multiple backends such as TensorFlow and Keras
* Very good for creating neural nets very quickly and hides awy a lot of tedious work
* Has been incorporated into official TensorFlow (which obviously only works with tensforflow)
%% Cell type:code id: tags:
``` python
# Say hello to keras
from keras.models import Sequential
from keras.layers import Dense, Activation
# Creating a model
model = Sequential()
# Adding layers to this model
# 1st Hidden layer
model.add(Dense(units=4, input_dim=4))
model.add(Activation("relu"))
# The output layer
model.add(Dense(units=1))
model.add(Activation("sigmoid"))
model.summary()
```
%% Output
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_7 (Dense) (None, 4) 20
_________________________________________________________________
activation_7 (Activation) (None, 4) 0
_________________________________________________________________
dense_8 (Dense) (None, 1) 5
_________________________________________________________________
activation_8 (Activation) (None, 1) 0
=================================================================
Total params: 25
Trainable params: 25
Non-trainable params: 0
_________________________________________________________________
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-17-0e94e6863fea> in <module>
1 # Say hello to keras
2
----> 3 from keras.models import Sequential
4 from keras.layers import Dense, Activation
5
ModuleNotFoundError: No module named 'keras'
%% Cell type:code id: tags:
``` python
# Fittind the model
```
%% Cell type:markdown id: tags:
## Network results on dataset used in previous notebooks
## Network Architecture
## CNN examples
%% Cell type:markdown id: tags:
TODO:
- does keras support scikit-learn api ? (.fit and .predict methods) ?
- if yes: we could use cross validation and hyper parameter optimzation for scikit-learn to evaluae / improve keras network.
%% Cell type:code id: tags:
``` python
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment