Commit 7d4e1bda authored by noureen.taj's avatar noureen.taj 🖖🏻

Update assignment1.md

parent 9420a93e
# Assignment 1 # Assignment 1
Hey there! Welcome to Knowledge Lens Intern Training Program.
This Assignment will serve as a quick refresher on the usage of NoSQL and Time-series databases.
There are three tasks in this assignment, on completion of which you'll learn:
* How to interact with Mongo DB
* Using Pandas Dataframe and generating your own excel reports
* Leveraging Kairos Time-series database for data ingestion and querying the same
* Publishing and Consuming messages via MQTT protocol
* Caching mechanism using Redis DB
Happy Coding! :tada:
## Task 1: Working with Mongo - Advanced ## :pushpin: Task 1: Working with Mongo - Advanced
## Areas covered: ### :golf: Areas covered:
- Timeseries Operation - Timeseries Operation
- Working with NoSQL - Working with NoSQL
- Working with Pandas - Working with Pandas
## Description: ### :books: Description:
You are given with a dataset of an inspection company in the form of a JSON file. The end goal of the project is to create an API interface that will provide the following: You are given with a dataset of an inspection company in the form of a JSON file. The end goal of the project is to create an API interface that will provide the following:
1. Business name with maximum number of "`Violation Issued`". 1. Business name with maximum number of "`Violation Issued`".
...@@ -38,31 +48,32 @@ Sample Document: ...@@ -38,31 +48,32 @@ Sample Document:
Bonus Points: Use Mongo Aggregate framework Bonus Points: Use Mongo Aggregate framework
### Tools to use: ### :wrench: Tools to use:
1. Pycharm / VSCode 1. Pycharm / VSCode
2. Robo3T / Studio3T / MongoDB Compass 2. Robo3T / Studio3T / MongoDB Compass
3. PyMongo 3. PyMongo
### Reference: ### :mag: References:
https://www.mongodb.com/docs/manual/tutorial/query-documents/ * [Querying Documents on Mongo](https://www.mongodb.com/docs/manual/tutorial/query-documents/)
https://www.mongodb.com/docs/manual/reference/operator/aggregation-pipeline/ * [Quick Summary on Mongo Aggregation Stages](https://www.mongodb.com/docs/manual/reference/operator/aggregation-pipeline/)
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_excel.html * [Generating Excel Sheets from a Pandas Dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_excel.html)
https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse * [How to return files on FastAPI response](https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse)
https://pymongo.readthedocs.io/en/stable/ * [PyMongo Official Documentation](https://pymongo.readthedocs.io/en/stable/)
_________________________________ _________________________________
## Task 2: Working with Timeseries ## :pushpin: Task 2: Working with Timeseries
## Areas covered: ### :golf: Areas covered:
- Timeseries Operation - Timeseries Operation
- Working with Timeseries - Working with Timeseries
- Working with Pandas - Working with Pandas
## Description: ### :books: Description:
You are given with a dataset of weather in the form of a CSV file. The end goal of the project is to create an API interface that will provide the following: You are given with a dataset of weather in the form of a CSV file. The end goal of the project is to create an API interface that will provide the following:
1. Get daily, monthly, weekly and monthly aggregate (min, max, and average) of the data and generate report in Excel format. 1. Get daily, monthly, weekly and monthly aggregate (min, max, and average) of the data and generate report in Excel format.
...@@ -74,20 +85,24 @@ Sample Document: ...@@ -74,20 +85,24 @@ Sample Document:
|10/03/2004 |18:00:00 |2.6 |1360 |150 |11.9 |1046 |166 |1056 |113 |1692 |1268 |13.6|48.9|0.7578| |10/03/2004 |18:00:00 |2.6 |1360 |150 |11.9 |1046 |166 |1056 |113 |1692 |1268 |13.6|48.9|0.7578|
### Tools to use: ### :wrench: Tools to use:
1. Pycharm / VSCode 1. Pycharm / VSCode
2. Pandas 2. Pandas
3. Kairos 3. Kairos
### Reference: ### :mag: References:
https://kairosdb.github.io/docs/restapi/QueryMetrics.html * [How to query Kairos DB using Metrics](https://kairosdb.github.io/docs/restapi/QueryMetrics.html)
https://pypi.org/project/kairosdb-python/
------------------------------------------------------ ------------------------------------------------------
## Task 3: Working with MQTT & REDIS ## :pushpin: Task 3: Working with MQTT & REDIS
### :golf: Areas covered:
- MQTT Protocol
- Caching using Redis DB
# Description ### :books: Description
Hospital with 'N' doctor's on duty, Hospital with 'N' doctor's on duty,
Receptionist need's to assign new patient's in round robin order. Receptionist need's to assign new patient's in round robin order.
...@@ -108,18 +123,15 @@ Patient details sample format: ...@@ -108,18 +123,15 @@ Patient details sample format:
Use Redis for caching information Use Redis for caching information
create a snippet which produces patient information to MQTT Create a snippet which produces patient information to MQTT.
create consumer's which consumes data from these topics and store to a redis db Create consumer's which consumes data from these topics and store to a redis db
### Tools to use: ### :wrench: Tools to use:
1. Pycharm / VSCode 1. Pycharm / VSCode
2. MQTT - (paho-mqtt) 2. MQTT - (PIP package: `paho-mqtt`)
3. REDIS - (redis) 3. REDIS - (PIP package: `redis`)
### Reference:
https://www.emqx.com/en/blog/how-to-use-mqtt-in-python
https://docs.redis.com/latest/rs/references/client_references/client_python/
### :mag: References:
* [Using MQTT in Python](https://www.emqx.com/en/blog/how-to-use-mqtt-in-python)
* [Connecting to Redis in Python](https://docs.redis.com/latest/rs/references/client_references/client_python/)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment