{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deepstream Test1 description\n",
    "--------------------------------------------------------------------------------------\n",
    "\n",
    "This sample creates instance of \"nvinfer\" element. Instance of\n",
    "the \"nvinfer\" uses TensorRT API to execute inferencing on a model. Using a\n",
    "correct configuration for a nvinfer element instance is therefore very\n",
    "important as considerable behaviors of the instance are parameterized\n",
    "through these configs.\n",
    "\n",
    "For reference, here are the config files used for this sample :\n",
    "The 4-class detector (referred to as pgie in this sample) uses\n",
    "    dstest1_pgie_config.txt\n",
    "\n",
    "In this sample, we first create one instance of \"nvinfer\", referred as the pgie.\n",
    "This is our 4 class detector and it detects for \"Vehicle , RoadSign, TwoWheeler,\n",
    "Person\".\n",
    "nvinfer element attach some MetaData to the buffer. By attaching\n",
    "the probe function at the end of the pipeline, one can extract meaningful\n",
    "information from this inference. Please refer the \"osd_sink_pad_buffer_probe\"\n",
    "function in the sample code. For details on the Metadata format, refer to the\n",
    "file \"gstnvdsmeta.h\""
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "[encoded data, 139016 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image.png](attachment:image.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Installation\n",
    "\n",
    "Prerequisites:\n",
    "\n",
    "    Ubuntu 18.04\n",
    "    DeepStream SDK 4.0.1 or later\n",
    "    Python 3\n",
    "    Gst Python v1.14.5\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GST Python installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#!sudo apt-get install python-gi-dev\n",
    "#!export GST_LIBS=\"-lgstreamer-1.0 -lgobject-2.0 -lglib-2.0\"\n",
    "#!export GST_CFLAGS=\"-pthread -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include\"\n",
    "#!git clone https://github.com/GStreamer/gst-python.git\n",
    "#!cd gst-python\n",
    "#!git checkout 1a8f48a\n",
    "#!./autogen.sh PYTHON=python3\n",
    "#!./configure PYTHON=python3\n",
    "#!make\n",
    "#!sudo make install\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download the Deepstream Python bindings file from here: \n",
    "\n",
    "https://developer.nvidia.com/deepstream_python_v0.5\n",
    " Untar the file in the following folder: tar xf ds_pybind_0.5.tbz2 -C <DeepStream 4.0.1 ROOT>/sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "!tar xf ds_pybind_0.5.tbz2 -C /home/nvidia/Downloads/deepstream/sources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------------------------------------------------------------------\n",
    "Package Contents\n",
    "--------------------------------------------------------------------------------\n",
    "The DeepStream Python package includes:\n",
    "1. Python bindings for DeepStream Metadata libraries\n",
    "   These bindings are provided as a compiled module, available for\n",
    "   x86_64 and Jetson platforms. Find them at:\n",
    "   bindings\n",
    "   \n",
    "                                                                                                                         |- x86_64\n",
    "            |- pyds.so\n",
    "       |- jetson\n",
    "            |- pyds.so\n",
    "2. DeepStream test apps in Python\n",
    "   Four test apps are available:\n",
    "   deepstream-test1/deepstream_test_1.py\n",
    "   deepstream-test2/deepstream_test_2.py\n",
    "   deepstream-test3/deepstream_test_3.py\n",
    "   deepstream-test3/deepstream_test_4.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run code:\n",
    "---------------------------------------------------------------------------------------------------\n",
    "\n",
    "Note - Make sure the folder path of where pyds.so file is located is in PATH.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../')\n",
    "sys.path.append('/home/nvidia/Downloads/deepstream/sources/python/bindings/x86_64')\n",
    "sys.path.append('/usr/lib/python3/dist-packages')\n",
    "from common.is_aarch_64 import is_aarch64\n",
    "import gi\n",
    "gi.require_version('Gst', '1.0')\n",
    "from gi.repository import GObject, Gst\n",
    "from common.is_aarch_64 import is_aarch64\n",
    "from common.bus_call import bus_call\n",
    "\n",
    "import pyds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Declaring class label ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PGIE_CLASS_ID_VEHICLE = 0\n",
    "PGIE_CLASS_ID_BICYCLE = 1\n",
    "PGIE_CLASS_ID_PERSON = 2\n",
    "PGIE_CLASS_ID_ROADSIGN = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def osd_sink_pad_buffer_probe(pad,info,u_data):\n",
    "    frame_number=0\n",
    "    #Intiallizing object counter with 0.\n",
    "    obj_counter = {\n",
    "        PGIE_CLASS_ID_VEHICLE:0,\n",
    "        PGIE_CLASS_ID_PERSON:0,\n",
    "        PGIE_CLASS_ID_BICYCLE:0,\n",
    "        PGIE_CLASS_ID_ROADSIGN:0\n",
    "    }\n",
    "    num_rects=0\n",
    "\n",
    "    gst_buffer = info.get_buffer()\n",
    "    if not gst_buffer:\n",
    "        print(\"Unable to get GstBuffer \")\n",
    "        return\n",
    "\n",
    "    # Retrieve batch metadata from the gst_buffer\n",
    "    # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the\n",
    "    # C address of gst_buffer as input, which is obtained with hash(gst_buffer)\n",
    "    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))\n",
    "    l_frame = batch_meta.frame_meta_list\n",
    "    while l_frame is not None:\n",
    "        try:\n",
    "            # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta\n",
    "            # The casting is done by pyds.glist_get_nvds_frame_meta()\n",
    "            # The casting also keeps ownership of the underlying memory\n",
    "            # in the C code, so the Python garbage collector will leave\n",
    "            # it alone.\n",
    "            frame_meta = pyds.glist_get_nvds_frame_meta(l_frame.data)\n",
    "        except StopIteration:\n",
    "            break\n",
    "\n",
    "        frame_number=frame_meta.frame_num\n",
    "        num_rects = frame_meta.num_obj_meta\n",
    "        l_obj=frame_meta.obj_meta_list\n",
    "        while l_obj is not None:\n",
    "            try:\n",
    "                # Casting l_obj.data to pyds.NvDsObjectMeta\n",
    "                obj_meta=pyds.glist_get_nvds_object_meta(l_obj.data)\n",
    "            except StopIteration:\n",
    "                break\n",
    "            obj_counter[obj_meta.class_id] += 1\n",
    "            try: \n",
    "                l_obj=l_obj.next\n",
    "            except StopIteration:\n",
    "                break\n",
    "\n",
    "        # Acquiring a display meta object. The memory ownership remains in\n",
    "        # the C code so downstream plugins can still access it. Otherwise\n",
    "        # the garbage collector will claim it when this probe function exits.\n",
    "        display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)\n",
    "        display_meta.num_labels = 1\n",
    "        py_nvosd_text_params = display_meta.text_params[0]\n",
    "        # Setting display text to be shown on screen\n",
    "        # Note that the pyds module allocates a buffer for the string, and the\n",
    "        # memory will not be claimed by the garbage collector.\n",
    "        # Reading the display_text field here will return the C address of the\n",
    "        # allocated string. Use pyds.get_string() to get the string content.\n",
    "        py_nvosd_text_params.display_text = \"Frame Number={} Number of Objects={} Vehicle_count={} Person_count={}\".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON])\n",
    "\n",
    "        # Now set the offsets where the string should appear\n",
    "        py_nvosd_text_params.x_offset = 10\n",
    "        py_nvosd_text_params.y_offset = 12\n",
    "\n",
    "        # Font , font-color and font-size\n",
    "        py_nvosd_text_params.font_params.font_name = \"Serif\"\n",
    "        py_nvosd_text_params.font_params.font_size = 10\n",
    "        # set(red, green, blue, alpha); set to White\n",
    "        py_nvosd_text_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0)\n",
    "\n",
    "        # Text background color\n",
    "        py_nvosd_text_params.set_bg_clr = 1\n",
    "        # set(red, green, blue, alpha); set to Black\n",
    "        py_nvosd_text_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0)\n",
    "        # Using pyds.get_string() to get display_text as string\n",
    "        print(pyds.get_string(py_nvosd_text_params.display_text))\n",
    "        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)\n",
    "        try:\n",
    "            l_frame=l_frame.next\n",
    "        except StopIteration:\n",
    "            break\n",
    "\t\t\t\n",
    "    return Gst.PadProbeReturn.OK\t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initializing GStreamer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard initialization procedure\n",
    "GObject.threads_init()\n",
    "Gst.init(None)\n",
    "\n",
    "# Create gstreamer elements\n",
    "# Create Pipeline element that will form a connection of other elements\n",
    "print(\"Creating Pipeline \\n \")\n",
    "pipeline = Gst.Pipeline()\n",
    "\n",
    "if not pipeline:\n",
    "    sys.stderr.write(\" Unable to create Pipeline \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating different elements of the stream"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a source element for reading from a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating Source \\n \")\n",
    "source = Gst.ElementFactory.make(\"filesrc\", \"file-source\")\n",
    "if not source:\n",
    "    sys.stderr.write(\" Unable to create Source \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a h264 parser as the input file is an elementary h264 stream"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating H264Parser \\n\")\n",
    "h264parser = Gst.ElementFactory.make(\"h264parse\", \"h264-parser\")\n",
    "if not h264parser:\n",
    "    sys.stderr.write(\" Unable to create h264 parser \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using nvdec_h264 for accelerated decoding on GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating Decoder \\n\")\n",
    "decoder = Gst.ElementFactory.make(\"nvv4l2decoder\", \"nvv4l2-decoder\")\n",
    "if not decoder:\n",
    "    sys.stderr.write(\" Unable to create Nvv4l2 Decoder \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a nvstreammux instance to form batches for one or more sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "streammux = Gst.ElementFactory.make(\"nvstreammux\", \"Stream-muxer\")\n",
    "if not streammux:\n",
    "    sys.stderr.write(\" Unable to create NvStreamMux \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up nvinfer to run inference on the decoders output\n",
    "### Note: Behaviour of inference is set through the config file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pgie = Gst.ElementFactory.make(\"nvinfer\", \"primary-inference\")\n",
    "if not pgie:\n",
    "    sys.stderr.write(\" Unable to create pgie \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a converter to convert from NV12 to RGBA as required by nvosd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nvvidconv = Gst.ElementFactory.make(\"nvvideoconvert\", \"convertor\")\n",
    "if not nvvidconv:\n",
    "    sys.stderr.write(\" Unable to create nvvidconv \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create OSD to draw on the converted RGBA buffer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nvosd = Gst.ElementFactory.make(\"nvdsosd\", \"onscreendisplay\")\n",
    "if not nvosd:\n",
    "    sys.stderr.write(\" Unable to create nvosd \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check required for jetson devices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if is_aarch64():\n",
    "    transform = Gst.ElementFactory.make(\"nvegltransform\", \"nvegl-transform\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finally render the osd output\n",
    "## Creating the necessary sink for the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating EGLSink \\n\")\n",
    "sink = Gst.ElementFactory.make(\"nveglglessink\", \"nvvideo-renderer\")\n",
    "if not sink:\n",
    "    sys.stderr.write(\" Unable to create egl sink \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setting PGIE and Streammux properties\n",
    "### Suggested reading: dstest1_pgie_config.txt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit this cell to change the inputs and the stream config\n",
    "print(\"Playing file /home/nvidia/Downloads/deepstream/sources/python/apps/deepstream-test1/sample_720p.h264 \")\n",
    "source.set_property('location', \\\n",
    "                    \"/home/nvidia/Downloads/deepstream/sources/python/apps/deepstream-test1/sample_720p.h264\")\n",
    "streammux.set_property('width', 1920)\n",
    "streammux.set_property('height', 1080)\n",
    "streammux.set_property('batch-size', 1)\n",
    "streammux.set_property('batched-push-timeout', 4000000)\n",
    "pgie.set_property('config-file-path', \\\n",
    "                  \"/home/nvidia/Downloads/deepstream/sources/python/apps/deepstream-test1/dstest1_pgie_config.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding all the above defined elements to a pipline and linking them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Adding elements to Pipeline \\n\")\n",
    "pipeline.add(source)\n",
    "pipeline.add(h264parser)\n",
    "pipeline.add(decoder)\n",
    "pipeline.add(streammux)\n",
    "pipeline.add(pgie)\n",
    "pipeline.add(nvvidconv)\n",
    "pipeline.add(nvosd)\n",
    "pipeline.add(sink)\n",
    "if is_aarch64():\n",
    "    pipeline.add(transform)\n",
    "# we link the elements together\n",
    "# file-source -> h264-parser -> nvh264-decoder ->\n",
    "# nvinfer -> nvvidconv -> nvosd -> video-renderer\n",
    "print(\"Linking elements in the Pipeline \\n\")\n",
    "source.link(h264parser)\n",
    "h264parser.link(decoder)\n",
    "\n",
    "sinkpad = streammux.get_request_pad(\"sink_0\")\n",
    "if not sinkpad:\n",
    "    sys.stderr.write(\" Unable to get the sink pad of streammux \\n\")\n",
    "srcpad = decoder.get_static_pad(\"src\")\n",
    "if not srcpad:\n",
    "    sys.stderr.write(\" Unable to get source pad of decoder \\n\")\n",
    "srcpad.link(sinkpad)\n",
    "streammux.link(pgie)\n",
    "pgie.link(nvvidconv)\n",
    "nvvidconv.link(nvosd)\n",
    "if is_aarch64():\n",
    "    nvosd.link(transform)\n",
    "    transform.link(sink)\n",
    "else:\n",
    "    nvosd.link(sink)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating an event loop and feeding bus messages to it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loop = GObject.MainLoop()\n",
    "bus = pipeline.get_bus()\n",
    "bus.add_signal_watch()\n",
    "bus.connect (\"message\", bus_call, loop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding a probe to get the meta data generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we add probe to the sink pad of the osd element, since by that time, \n",
    "# the buffer would have had got all the metadata.\n",
    "osdsinkpad = nvosd.get_static_pad(\"sink\")\n",
    "if not osdsinkpad:\n",
    "    sys.stderr.write(\" Unable to get sink pad of nvosd \\n\")\n",
    "\n",
    "osdsinkpad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Starting the pipline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Starting pipeline \\n\")\n",
    "pipeline.set_state(Gst.State.PLAYING)\n",
    "try:\n",
    "    loop.run()\n",
    "except:\n",
    "    pass\n",
    "# cleaning up as the pipeline comes to an end\n",
    "pipeline.set_state(Gst.State.NULL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# To setting up inferencing interval:\n",
    "    0: inference every frame\n",
    "    1: inference every other frame (e.g. detect objects every other frame, use tracker to localize in-between)\n",
    "    2: inference every third frame\n",
    "\n",
    "\n",
    "Go to the config file mentioned below \n",
    "\n",
    "\"/home/nvidia/Downloads/deepstream/sources/python/apps/deepstream-test1/dstest1_pgie_config.txt\"\n",
    "\n",
    "and change the interval as required. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting static image of the display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 2692864 characters]\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='ds_test1.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
