Skip to content

P
preprocess-data
Commit e82a863d authored by dasharatha.vamshi's avatar dasharatha.vamshi
Browse files
Options

pkl

parent 910da253
Show whitespace changes
Inline Side-by-side
Showing with 12 additions and 79 deletions
logs/PreProccessComponent.log deleted 100644 → 0
View file @ 910da253
2021-02-23 17:56:17,606 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 17:56:17,606 INFO PreProccessComponent Reading Json file
2021-02-23 17:56:17,622 INFO PreProccessComponent Parsing the pickle file
2021-02-23 17:59:10,742 INFO PreProccessComponent Reading Json file
2021-02-23 17:59:10,760 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:01:29,652 INFO PreProccessComponent Reading Json file
2021-02-23 18:01:29,671 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:02:08,816 INFO PreProccessComponent Reading Json file
2021-02-23 18:02:08,827 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:02:40,875 INFO PreProccessComponent Reading Json file
2021-02-23 18:02:40,887 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:04:47,559 INFO PreProccessComponent Reading Json file
2021-02-23 18:04:47,570 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:06:03,437 INFO PreProccessComponent Reading Json file
2021-02-23 18:06:03,448 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:06:26,273 INFO PreProccessComponent Reading Json file
2021-02-23 18:06:26,282 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:06:49,400 INFO PreProccessComponent Reading Json file
2021-02-23 18:06:49,412 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:07:24,418 INFO PreProccessComponent Reading Json file
2021-02-23 18:07:24,430 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:08:25,242 INFO PreProccessComponent Reading Json file
2021-02-23 18:08:25,253 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:09:21,818 INFO PreProccessComponent Reading Json file
2021-02-23 18:09:21,830 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:12:27,419 INFO PreProccessComponent Reading Json file
2021-02-23 18:12:27,438 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:12:27,444 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 18:12:27,445 INFO PreProccessComponent Component executed Successfully
2021-02-23 18:13:15,958 INFO PreProccessComponent Reading Json file
2021-02-23 18:13:15,978 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:13:15,986 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 18:13:15,991 INFO PreProccessComponent Component executed Successfully
2021-02-23 18:13:32,970 INFO PreProccessComponent Reading Json file
2021-02-23 18:13:32,985 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:13:32,993 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 18:13:32,994 INFO PreProccessComponent Component executed Successfully
2021-02-23 18:14:52,633 INFO PreProccessComponent Reading Json file
2021-02-23 18:14:52,645 INFO PreProccessComponent Parsing the pickle file
2021-02-23 18:14:52,650 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 18:14:52,650 INFO PreProccessComponent Component executed Successfully
2021-02-23 19:02:28,341 INFO PreProccessComponent Reading Json file
2021-02-23 19:02:28,351 INFO PreProccessComponent Parsing the pickle file
2021-02-23 19:02:29,640 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 19:02:29,641 INFO PreProccessComponent Component executed Successfully
2021-02-23 19:03:02,447 INFO PreProccessComponent Reading Json file
2021-02-23 19:03:02,457 INFO PreProccessComponent Parsing the pickle file
2021-02-23 19:03:14,509 INFO PreProccessComponent Reading Json file
2021-02-23 19:03:14,519 INFO PreProccessComponent Parsing the pickle file
2021-02-23 19:03:15,704 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 19:03:15,705 INFO PreProccessComponent Component executed Successfully
2021-02-23 20:01:03,806 INFO PreProccessComponent Reading Json file
2021-02-23 20:01:03,816 INFO PreProccessComponent Parsing the pickle file
2021-02-23 20:01:10,052 INFO PreProccessComponent Reading Json file
2021-02-23 20:01:10,063 INFO PreProccessComponent Parsing the pickle file
2021-02-23 20:01:11,229 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 20:01:11,230 INFO PreProccessComponent Component executed Successfully
2021-02-23 20:01:18,361 INFO PreProccessComponent Reading Json file
2021-02-23 20:01:18,371 INFO PreProccessComponent Parsing the pickle file
2021-02-23 20:01:19,794 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 20:01:19,795 INFO PreProccessComponent Component executed Successfully
2021-02-23 20:01:31,699 INFO PreProccessComponent Reading Json file
2021-02-23 20:01:31,708 INFO PreProccessComponent Parsing the pickle file
2021-02-23 20:01:32,896 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 20:01:32,896 INFO PreProccessComponent Component executed Successfully
2021-02-23 20:04:25,902 INFO PreProccessComponent Reading Json file
2021-02-23 20:04:25,913 INFO PreProccessComponent Parsing the pickle file
2021-02-23 20:04:27,159 INFO PreProccessComponent Got the data writing it to pickle file (preprocessed_X.pkl)
2021-02-23 20:04:27,162 INFO PreProccessComponent Component executed Successfully
2021-02-23 20:56:37,844 INFO PreProccessComponent {'shared_volume': 'test', 'json_path': 'E:\\welspun-defects\\preprocess_data-ilendev\\response.json', 'fillna_method': 'ffill', 'standard_scalar_path': ''}
2021-02-23 20:56:37,844 INFO PreProccessComponent Reading Json file
2021-02-23 20:56:37,855 INFO PreProccessComponent Parsing the pickle file at location E:\iLens-AI\Git\preprocess-data\StandardScaler.pkl ......
2021-02-23 20:56:39,016 INFO PreProccessComponent Got the data writing it to pickle files (preprocessed_X.pkl and date_X.pkl)
2021-02-23 20:56:39,018 INFO PreProccessComponent Component executed Successfully
main.py
View file @ e82a863d
......@@ -13,13 +13,20 @@ class PreProcessComponent:
self.ds, self.tempmax, self.tempmin, self.temp, self.humidity, self.precip, self.winddir, self.cloudcover, self.visibility, self.date, self.windspeed = ([] for i in range(11))
def read_pickle_file(self, file):
try:
logger.info("Parsing the pickle file at location " + file + " ......")
return pickle.load(open(file, "rb"))
except:
logger.info("Failed to read pickle file")
def preprocess(self, forcast_json, standard_scalar_pkl_path, null_value_method):
try:
logger.info("Reading Json file")
with open(forcast_json) as f:
data = json.load(f)
except:
logger.info("Failed reading Json File")
for i in data[PreProcessConstants.DAYS_KEY]:
self.date.append(i[PreProcessConstants.DATETIME_KEY])
self.ds.append(int(i[PreProcessConstants.DATETIME_KEY].split("-")[1]))
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment