{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/tonegas/nnodely/blob/main/examples/linear.ipynb\" target=\"_blank\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\">\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Layer\n",
    "\n",
    "Represents a Linear relation in the neural network model.\n",
    "\n",
    "The Linear relation works along the input dimension (third dimension) of the input tensor.\n",
    "\n",
    "The Linear relation can be instantiated with the following modalities:\n",
    "\n",
    "1- Setting and initialize the weight vector\n",
    "\n",
    "2- Setting and initialize the bias vector\n",
    "\n",
    "3- Adding Dropout regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>-- nnodely_v1.5.0 --<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<\n"
     ]
    }
   ],
   "source": [
    "# uncomment the command below to install the nnodely package\n",
    "#!pip install nnodely\n",
    "\n",
    "from nnodely import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Usage\n",
    "\n",
    "Create a Linear Relation block with random Weights initialization and 5 outputs. \n",
    "\n",
    "(By default, there is no bias. to add the bias parameter set the 'b' attribute to True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = Input('x').tw(0.05)\n",
    "linear = Linear(output_dimension=5)(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Passing a Parameter\n",
    "\n",
    "Create a Fir Relation block and setting the weight and bias with a pre-defined parameter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[check_names] The name 'x' is already in defined as NeuObj but it is overwritten.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "x = Input('x').last()\n",
    "\n",
    "weight = Parameter('W', values=[[1]])\n",
    "bias = Parameter('b', values=[1])\n",
    "\n",
    "linear = Linear(W=weight, b=bias)(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization functions\n",
    "\n",
    "Create a Fir Relation block and initialize the weight and bias using an initialization function.\n",
    "\n",
    "Set the argument of the initialization function using a dictionary as shown below.\n",
    "\n",
    "(you can find all the initialization function inside the 'initializer' module. You can also define your own initialization function!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[check_names] The name 'x' is already in defined as NeuObj but it is overwritten.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "x = Input('x').last()\n",
    "linear = Linear(output_dimension=5, b=True, W_init=init_negexp, b_init=init_constant, b_init_params={'value':1})(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout\n",
    "\n",
    "Create a Linear Relation block with 10 outputs and dropout regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[check_names] The name 'x' is already in defined as NeuObj but it is overwritten.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "input = Input('x')\n",
    "linear = Linear(output_dimension=10, dropout=0.2)(input.sw(2))\n",
    "output = Output('out', linear)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv (3.11.4)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}