Chapter 3 ColabNotebook prepare_model.ipynb getting errors

[jomoengineer]

I am seeing errors when running the Method 1 prepare_model.ipynb.
With Pandas 1.5.x I see this:

Retrieving weather data for canazel

Currently retrieving data for canazel: from 2011-01-01 to 2011-01-31
/home/jomopi/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/wwo_hist/init.py:75: FutureWarning: Argument closed is deprecated in favor of inclusive.
list_mon_end = pd.date_range(start_date, end_date, freq='M', closed='left')

KeyError Traceback (most recent call last)
Cell In[5], line 6
3 location_list = ['canazel']
5 # retrieve_hist_data returns a list of dataframe(s)
----> 6 hist_df = retrieve_hist_data(api_key,
7 location_list,
8 '01-JAN-2011',
9 '31-DEC-2020',
10 frequency,
11 location_label = False,
12 export_csv = False,
13 store_df = True)
15 # Extract temperature, humidity and precipitation
16 t_list = hist_df[0].tempC.astype(float).to_list()

File ~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/wwo_hist/init.py:120, in retrieve_hist_data(api_key, location_list, start_date, end_date, frequency, location_label, export_csv, store_df, response_cache_path)
118 for location in location_list:
119 print('\n\nRetrieving weather data for ' + location + '\n\n')
--> 120 df_this_city = retrieve_this_location(api_key, location, start_date, end_date, frequency, response_cache_path)
122 if (location_label == True):
123 # add city name as prefix to the colnames
124 df_this_city = df_this_city.add_prefix(location + '_')

File ~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/wwo_hist/init.py:102, in retrieve_this_location(api_key, location, start_date, end_date, frequency, response_cache_path)
100 with open(f'{response_cache_path}/{location}{start_d}{end_d}', 'w') as f:
101 json.dump(json_data, f)
--> 102 data = json_data['data']['weather']
103 # call function to extract json object
104 df_this_month = extract_monthly_data(data)

KeyError: 'weather'

Then at :"Convert the model to TensorFlow Lite file format (FlatBuffers) as save it as .tflite", I get this.

2024-02-13 18:17:08.250194: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:378] Ignored output_format.
2024-02-13 18:17:08.250234: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:381] Ignored drop_control_dependency.
2024-02-13 18:17:08.250693: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: snow_forecast
2024-02-13 18:17:08.251821: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve }
2024-02-13 18:17:08.251852: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: snow_forecast
2024-02-13 18:17:08.253947: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled
2024-02-13 18:17:08.255143: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle.
2024-02-13 18:17:08.324753: I tensorflow/cc/saved_model/loader.cc:217] Running initialization op on SavedModel bundle at path: snow_forecast
2024-02-13 18:17:08.345514: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 94824 microseconds.
2024-02-13 18:17:08.372504: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var MLIR_CRASH_REPRODUCER_DIRECTORY to enable.
Summary on the non-converted ops:

• Accepted dialects: tfl, builtin, func
• Non-Converted Ops: 4, Total Ops 10, % non-converted = 40.00 %
• 4 ARITH ops

• arith.constant: 4 occurrences (f32: 4)

  (f32: 2)
  (f32: 1)
  fully_quantize: 0, inference_type: 6WARNING:absl:Buffer deduplication procedure will be skipped when flatbuffer library is not properly loaded
  , input_inference_type: INT8, output_inference_type: INT8

---

TypeError Traceback (most recent call last)
/tmp/ipykernel_105153/1373129489.py in ?()
----> 1 tflite_model_quant = converter.convert()
2 open(TFL_MODEL_FILE, "wb").write(tflite_model_quant)

~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/tensorflow/lite/python/lite.py in ?(self, *args, **kwargs)
1136 @functools.wraps(convert_func)
1137 def wrapper(self, *args, **kwargs):
1138 # pylint: disable=protected-access
-> 1139 return self._convert_and_export_metrics(convert_func, *args, **kwargs)
1140 # pylint: enable=protected-access

~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/tensorflow/lite/python/lite.py in ?(self, convert_func, *args, **kwargs)
1107 # TODO(b/202090541): Collects sparsity block size information.
1108 sparsity_modes = _get_sparsity_modes(model_object)
1109 self._metadata.options.modelOptimizationModes.extend(sparsity_modes)
1110 model_object = _populate_conversion_metadata(model_object, self._metadata)
-> 1111 return flatbuffer_utils.convert_object_to_bytearray(model_object)

~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/tensorflow/lite/tools/flatbuffer_utils.py in ?(model_object, extra_buffer)
89 def convert_object_to_bytearray(model_object, extra_buffer=b''):
90 """Converts a tflite model from an object to a immutable bytearray."""
91 # Initial size of the buffer, which will grow automatically if needed
92 builder = flatbuffers.Builder(1024)
---> 93 model_offset = model_object.Pack(builder)
94 builder.Finish(model_offset, file_identifier=_TFLITE_FILE_IDENTIFIER)
95 model_bytearray = bytes(builder.Output())
96 model_bytearray = model_bytearray + extra_buffer

~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/tensorflow/lite/python/schema_py_generated.py in ?(self, builder)
18097 operatorCodeslist.append(self.operatorCodes[i].Pack(builder))
18098 ModelStartOperatorCodesVector(builder, len(self.operatorCodes))
18099 for i in reversed(range(len(self.operatorCodes))):
18100 builder.PrependUOffsetTRelative(operatorCodeslist[i])

18101 operatorCodes = builder.EndVector()
18102 if self.subgraphs is not None:
18103 subgraphslist = []
18104 for i in range(len(self.subgraphs)):

TypeError: Builder.EndVector() missing 1 required positional argument: 'vectorNumElems'

[jomoengineer]

This is what I see with Pandas 2.x:

Retrieving weather data for canazel


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 6
      3 location_list = ['canazel']
      5 # retrieve_hist_data returns a list of dataframe(s)
----> 6 hist_df = retrieve_hist_data(api_key,
      7                              location_list,
      8                              '01-JAN-2011',
      9                              '31-DEC-2020',
     10                              frequency,
     11                              location_label = False,
     12                              export_csv = False,
     13                              store_df = True)
     15 # Extract temperature, humidity and precipitation
     16 t_list = hist_df[0].tempC.astype(float).to_list()

File ~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/wwo_hist/__init__.py:120, in retrieve_hist_data(api_key, location_list, start_date, end_date, frequency, location_label, export_csv, store_df, response_cache_path)
    118 for location in location_list:
    119     print('\n\nRetrieving weather data for ' + location + '\n\n')
--> 120     df_this_city = retrieve_this_location(api_key, location, start_date, end_date, frequency, response_cache_path)
    122     if (location_label == True):
    123         # add city name as prefix to the colnames
    124         df_this_city = df_this_city.add_prefix(location + '_')

File ~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/wwo_hist/__init__.py:75, in retrieve_this_location(api_key, location, start_date, end_date, frequency, response_cache_path)
     72 list_mon_begin = pd.concat([pd.Series(pd.to_datetime(start_date)), pd.Series(list_mon_begin)], ignore_index=True)
     74 # create list of month end dates for range between start and end dates non-inclusive (open)
---> 75 list_mon_end = pd.date_range(start_date, end_date, freq='M', closed='left')
     76 # convert to Series and add end_date at end
     77 list_mon_end = pd.concat([pd.Series(list_mon_end), pd.Series(pd.to_datetime(end_date))], ignore_index=True)

File ~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/pandas/core/indexes/datetimes.py:1008, in date_range(start, end, periods, freq, tz, normalize, name, inclusive, unit, **kwargs)
   1005 if freq is None and com.any_none(periods, start, end):
   1006     freq = "D"
-> 1008 dtarr = DatetimeArray._generate_range(
   1009     start=start,
   1010     end=end,
   1011     periods=periods,
   1012     freq=freq,
   1013     tz=tz,
   1014     normalize=normalize,
   1015     inclusive=inclusive,
   1016     unit=unit,
   1017     **kwargs,
   1018 )
   1019 return DatetimeIndex._simple_new(dtarr, name=name)

TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'

[jomoengineer]

I was able to get passed the Pandas 2.x issue by making changes in WorldWeatherOnline.

ekapope/WorldWeatherOnline#22

However, I am still seeing an issue with KeyError 'weather'.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[5], line 6
      3 location_list = ['canazel']
      5 # retrieve_hist_data returns a list of dataframe(s)
----> 6 hist_df = retrieve_hist_data(api_key,
      7                              location_list,
      8                              '01-JAN-2011',
      9                              '31-DEC-2020',
     10                              frequency,
     11                              location_label = False,
     12                              export_csv = False,
     13                              store_df = True)
     15 # Extract temperature, humidity and precipitation
     16 t_list = hist_df[0].tempC.astype(float).to_list()

File ~/develop/machine_leaning/WorldWeatherOnline/wwo_hist/__init__.py:119, in retrieve_hist_data(api_key, location_list, start_date, end_date, frequency, location_label, export_csv, store_df, response_cache_path)
    117 for location in location_list:
    118     print('\n\nRetrieving weather data for ' + location + '\n\n')
--> 119     df_this_city = retrieve_this_location(api_key, location, start_date, end_date, frequency, response_cache_path)
    121     if (location_label == True):
    122         # add city name as prefix to the colnames
    123         df_this_city = df_this_city.add_prefix(location + '_')

File ~/develop/machine_leaning/WorldWeatherOnline/wwo_hist/__init__.py:101, in retrieve_this_location(api_key, location, start_date, end_date, frequency, response_cache_path)
     99     with open(f'{response_cache_path}/{location}_{start_d}_{end_d}', 'w') as f:
    100         json.dump(json_data, f)
--> 101 data = json_data['data']['weather']
    102 # call function to extract json object
    103 df_this_month = extract_monthly_data(data)

KeyError: 'weather'

[jomoengineer]

Looks like the 'weather' error was due to me misspelling the name 'canazei'. Method 1 works now after making the changes to 'wwo_hist' with changing 'close' to 'inclusive' There are still some deprecated warnings though.

However, I am still seeing the 'vectorNumElems' error in 'tensorflow'.

2024-02-14 22:56:05.435587: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:378] Ignored output_format.
2024-02-14 22:56:05.435642: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:381] Ignored drop_control_dependency.
2024-02-14 22:56:05.436237: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: snow_forecast
2024-02-14 22:56:05.437412: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve }
2024-02-14 22:56:05.437455: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: snow_forecast
2024-02-14 22:56:05.439953: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled
2024-02-14 22:56:05.440956: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle.
2024-02-14 22:56:05.479323: I tensorflow/cc/saved_model/loader.cc:217] Running initialization op on SavedModel bundle at path: snow_forecast
2024-02-14 22:56:05.491198: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 54967 microseconds.
2024-02-14 22:56:05.501956: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
Summary on the non-converted ops:
---------------------------------
 * Accepted dialects: tfl, builtin, func
 * Non-Converted Ops: 4, Total Ops 10, % non-converted = 40.00 %
 * 4 ARITH ops

- arith.constant:    4 occurrences  (f32: 4)



  (f32: 2)
  (f32: 1)
fully_quantize: 0, inference_type: 6, input_inference_type: INT8, output_inference_type: INT8
WARNING:absl:Buffer deduplication procedure will be skipped when flatbuffer library is not properly loaded
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_47203/1373129489.py in ?()
----> 1 tflite_model_quant = converter.convert()
      2 open(TFL_MODEL_FILE, "wb").write(tflite_model_quant)

~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/tensorflow/lite/python/lite.py in ?(self, *args, **kwargs)
   1136   @functools.wraps(convert_func)
   1137   def wrapper(self, *args, **kwargs):
   1138     # pylint: disable=protected-access
-> 1139     return self._convert_and_export_metrics(convert_func, *args, **kwargs)
   1140     # pylint: enable=protected-access

~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/tensorflow/lite/python/lite.py in ?(self, convert_func, *args, **kwargs)
   1107     # TODO(b/202090541): Collects sparsity block size information.
   1108     sparsity_modes = _get_sparsity_modes(model_object)
   1109     self._metadata.options.modelOptimizationModes.extend(sparsity_modes)
   1110     model_object = _populate_conversion_metadata(model_object, self._metadata)
-> 1111     return flatbuffer_utils.convert_object_to_bytearray(model_object)

~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/tensorflow/lite/tools/flatbuffer_utils.py in ?(model_object, extra_buffer)
     89 def convert_object_to_bytearray(model_object, extra_buffer=b''):
     90   """Converts a tflite model from an object to a immutable bytearray."""
     91   # Initial size of the buffer, which will grow automatically if needed
     92   builder = flatbuffers.Builder(1024)
---> 93   model_offset = model_object.Pack(builder)
     94   builder.Finish(model_offset, file_identifier=_TFLITE_FILE_IDENTIFIER)
     95   model_bytearray = bytes(builder.Output())
     96   model_bytearray = model_bytearray + extra_buffer

~/develop/machine_leaning/ml_testing/lib/python3.11/site-packages/tensorflow/lite/python/schema_py_generated.py in ?(self, builder)
  18097                 operatorCodeslist.append(self.operatorCodes[i].Pack(builder))
  18098             ModelStartOperatorCodesVector(builder, len(self.operatorCodes))
  18099             for i in reversed(range(len(self.operatorCodes))):
  18100                 builder.PrependUOffsetTRelative(operatorCodeslist[i])
> 18101             operatorCodes = builder.EndVector()
  18102         if self.subgraphs is not None:
  18103             subgraphslist = []
  18104             for i in range(len(self.subgraphs)):

TypeError: Builder.EndVector() missing 1 required positional argument: 'vectorNumElems'

[jomoengineer]

I was able to get passed the TypeError for 'vectorNumElems' buy downgrading flatbuffers to ver 23.5.26.
Ex:
pip3 uninstall flatbuffers
pip3 install flatbuffers=23.5.25

There appears to be a conflict with Tensorflow 2.15.0.

I was able to run the complete notebook on a Raspberry 5 running Bookworm.

[adesolasamuel]

I was able to solve the "TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'" error by opening the wwo_hist module file and editing the said file "python3.11/site-packages/wwo_hist/init.py" by changing the close keyword to inclusive, there are 2 places where closed were used so I changed the two to inclusive and everything worked well.

I did a full video on this particular chapter here: https://youtu.be/ip8nqTWu_Mk?si=MZhW7HQszf1A7iMn

Credits to the TinyML Cookbook