Implementing SQLite based request queue

* the husk * setPath fix * Co-authored-by: ArtursKadikis <[email protected]> * Reworking storage module API and creating implementation for the memory variant (#111) * - polymorphous behavior added - sqlite storge class added - code is building * storage modules integration stable state * - log added * storage unit tests added * unit tests updated * - virtual destructor - sqlite storage tests * files and class names updated * - storage unit tests improved - PR Changes - Code formatting * Log print updated while adding event. * cmakelist file updated * - Abstract methods signatures updated - Stable state * - new methods added - Unit test improved - stable state * - Unit tests add/updated - methods signatures improved - Fixed potential bugs * - logic updated * - Fixed issue with storage moduel - Unit tests * - request remove front code refactored * - Moved to new structure * -Unit tests fixed -Updated logic * memory deallocated * - Pointer to smart pointer * - RQpeekFront would return DataENtry with id = -1 and data = "" - Incremental Id Unit tests added * - make_shard used - peekAll and clearAll * removed logs * storage module unit tests updated * - unit test changes, * - Only unit tests added Note: Code isn't compiling * - unit test comments added - compile errors fixed - All unit tests are doing fine. * Code refactor * PR Changes, * test changes, * - missed requested chagnes * - RQ removed on the basis of id, not ref * Storage module methods documentation added. * request processing is functional * Method comments updated * - fixed unit tests * - fix bug - views test running. * - fixe a potential bug, --------- Co-authored-by: Zahid Zafar <> * View changes added (#113) * views * comments * extracted things (#114) * bits * Co-authored-by: ArtursKadikis <[email protected]> * Adding request queue db variant (#112) * - Request sql calls added * - code compile checks - storage unit tests logic updated * storage tests reworked * - sqlite storage unit tests fixed * - clearSDK - debugEQState - Views tests * - finishing test * - PR Requested changes - raw pointer to smart pointer * Requested changes * - crash tests fixed - sessions tests fixed * - set request queue size - safe check on configuration values - new configuration tests * Request module request threshold unit tests added. * - safety check for storage modules * - unit tests added in the storage module - PR changes - fixed event schema creation * Changelog added SDK Version updated * PR Changes * Update views.cpp * comments and name changes * Co-authored-by: ArtursKadikis <[email protected]> * new vacuum * its a me function --------- Co-authored-by: Zahid Zafar <> Co-authored-by: ArtursK <[email protected]> Co-authored-by: turtledreams <[email protected]> * Update CHANGELOG.md * Update CHANGELOG.md * database --------- Co-authored-by: Zahid Zafar <[email protected]> Co-authored-by: ArtursK <[email protected]>
Countly · Mar 10, 2023 · 0d34011 · 0d34011
1 parent e978b57
commit 0d34011
Show file tree

Hide file tree

Showing 27 changed files with 1,908 additions and 80 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,8 @@
+22.09.0
+* ! Minor breaking change ! SDK configuration now can't be changed after initialization/start
+* Added a persistent requests queue when building the SDK with the 'COUNTLY_USE_SQLITE' flag
+* Fixed a bug where view's name was being overriden by segmentation provided.
+
 22.06.4
 * Fixed a bug where the SDK 'mutex' was being locked twice when built with the 'COUNTLY_USE_SQLITE' flag.
 

diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -36,8 +36,6 @@ set(COUNTLY_PUBLIC_HEADERS
 
   ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/logger_module.hpp
   ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/crash_module.hpp
-  ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/request_module.hpp
-  ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/request_builder.hpp
   ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/views_module.hpp)
 
 add_library(countly
@@ -48,6 +46,8 @@ add_library(countly
   ${CMAKE_CURRENT_SOURCE_DIR}/src/request_module.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/src/crash_module.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/src/request_builder.cpp
+  ${CMAKE_CURRENT_SOURCE_DIR}/src/storage_module_db.cpp
+  ${CMAKE_CURRENT_SOURCE_DIR}/src/storage_module_memory.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/src/event.cpp)
 
 target_include_directories(countly
@@ -107,8 +107,10 @@ if(COUNTLY_BUILD_TESTS)
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/main.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/views.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/session.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/tests/storage.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/event.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/crash.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/tests/request.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/config.cpp)
 
     target_compile_definitions(countly-tests PRIVATE COUNTLY_BUILD_TESTS)

diff --git a/bin/README.md b/bin/README.md
@@ -0,0 +1,25 @@
+## Build Scripts
+
+Scripts provided here offers a convenient way to build sample app and the tests at different platforms.
+Still SQLite setup must be done separately at the "/vendor/sqlite" directory if SQLite is needed.
+Current cmake options set in the script that differs from the default are:
+
+- COUNTLY_BUILD_SAMPLE = ON
+- COUNTLY_BUILD_TESTS = ON
+
+To change any other options you can manually modify the scripts.
+
+### Run
+
+You can run the scripts simply by executing them from the root directory.
+
+### Flow
+
+When run the script would:
+
+- Check if a build folder is present at the root and erase it if it exists
+- Run Cmake with mentioned options
+- Build countly-sample app
+- Build countly-test
+
+After this point you can just go to the "/build" directory and execute those build files
diff --git a/bin/mac_build_script.sh b/bin/mac_build_script.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+
+if [ "${PWD##*/}" = "build" ]
+then
+    echo "In build folder. Changing directory..."
+    cd ..
+    echo "Current directory: $PWD"
+else
+    echo "Not in build folder. Current directory: $PWD"
+fi
+
+echo "Deleting folder and its subfolders..."
+rm -rf build
+echo "Folder and subfolders deleted."
+
+cmake -DCOUNTLY_BUILD_SAMPLE=1 -DCOUNTLY_BUILD_TESTS=1 -DCOUNTLY_USE_CUSTOM_SHA256=OFF -DCOUNTLY_USE_SQLITE=OFF -B build .
+
+cd build
+make ./countly-sample
+make ./countly-tests
diff --git a/bin/windows_build_script.bat b/bin/windows_build_script.bat
@@ -0,0 +1,15 @@
+@echo off
+if /i "%CD:~-5%"=="build" (
+    echo In build folder. Changing directory...
+    cd ..
+    echo Current directory: %CD%
+) else (
+    echo Not in build folder. Current directory: %CD%
+)
+echo Deleting folder and its subfolders...
+rmdir /s /q build
+echo Folder and subfolders deleted.
+cmake -DCOUNTLY_BUILD_SAMPLE=1 -DCOUNTLY_BUILD_TESTS=1 -DCOUNTLY_USE_CUSTOM_SHA256=OFF -DCOUNTLY_USE_SQLITE=OFF -B build . -G "Unix Makefiles"
+cd build
+make ./countly-sample
+make ./countly-tests
diff --git a/examples/example_integration.cpp b/examples/example_integration.cpp
@@ -34,10 +34,10 @@ int main() {
   cout << "Sample App" << endl;
   Countly &ct = Countly::getInstance();
   ct.alwaysUsePost(true);
+  ct.setLogger(printLog);
+  ct.SetPath("databaseFileName.db"); // this will be only built into account if the correct configurations are set
   ct.setDeviceID("test-device-id");
   // ct.setSalt("test-salt");
-
-  ct.setLogger(printLog);
   // OS, OS_version, device, resolution, carrier, app_version);
   ct.SetMetrics("Windows 10", "10.22", "Mac", "800x600", "Carrier", "1.0");
   // Server and port

diff --git a/include/countly.hpp b/include/countly.hpp
@@ -22,17 +22,20 @@
 #endif
 #include "countly/event.hpp"
 #include "countly/logger_module.hpp"
+#include "countly/storage_module_base.hpp"
 #include "countly/views_module.hpp"
+#include <countly/crash_module.hpp>
 #include <countly/request_builder.hpp>
 #include <countly/request_module.hpp>
-#include <countly/crash_module.hpp>
 
 namespace cly {
 class Countly : public cly::CountlyDelegates {
 public:
   Countly();
 
   ~Countly();
+
+  // Returns the singleton instance of Countly
   static Countly &getInstance();
 
   // Do not implicitly generate the copy constructor, this is a singleton.
@@ -43,6 +46,8 @@ class Countly : public cly::CountlyDelegates {
 
   void alwaysUsePost(bool value);
 
+  void setMaxRequestQueueSize(unsigned int requestQueueSize);
+
   void setSalt(const std::string &value);
 
   void setLogger(void (*fun)(LogLevel level, const std::string &message));
@@ -93,6 +98,8 @@ class Countly : public cly::CountlyDelegates {
 
   void addEvent(const cly::Event &event);
 
+  void addEventToSqlite(const cly::Event &event);
+
   void setMaxEvents(size_t value);
 
   void flushEvents(std::chrono::seconds timeout = std::chrono::seconds(30));
@@ -179,7 +186,14 @@ class Countly : public cly::CountlyDelegates {
   }
 
   /* Provide 'updateInterval' in seconds. */
-  inline void setAutomaticSessionUpdateInterval(unsigned short updateInterval) { configuration->sessionDuration = updateInterval; }
+  inline void setAutomaticSessionUpdateInterval(unsigned short updateInterval) {
+    if (is_sdk_initialized) {
+      log(LogLevel::WARNING, "[Countly][setAutomaticSessionUpdateInterval] You can not set the session duration after SDK initialization.");
+      return;
+    }
+
+    configuration->sessionDuration = updateInterval;
+  }
 
 #ifdef COUNTLY_BUILD_TESTS
   /**
@@ -188,15 +202,7 @@ class Countly : public cly::CountlyDelegates {
    * You should not be using this method.
    * @return a vector object containing events.
    */
-  const std::vector<std::string> debugReturnStateOfEQ() {
-
-#ifdef COUNTLY_USE_SQLITE
-    return {};
-#else
-    std::vector<std::string> v(event_queue.begin(), event_queue.end());
-    return v;
-#endif
-  }
+  std::vector<std::string> debugReturnStateOfEQ();
 
   /**
    * This function should not be used as it will be removed in a future release.
@@ -229,6 +235,9 @@ class Countly : public cly::CountlyDelegates {
   void _deleteThread();
   void _sendIndependantLocationRequest();
   void log(LogLevel level, const std::string &message);
+#ifdef COUNTLY_USE_SQLITE
+  bool createEventTableSchema();
+#endif
 
   /**
    * Helper methods to fetch remote config from the server.
@@ -257,11 +266,12 @@ class Countly : public cly::CountlyDelegates {
   std::unique_ptr<std::thread> thread;
   std::unique_ptr<cly::CrashModule> crash_module;
   std::unique_ptr<cly::ViewsModule> views_module;
+
   std::shared_ptr<cly::CountlyConfiguration> configuration;
   std::shared_ptr<cly::LoggerModule> logger;
-
   std::shared_ptr<cly::RequestBuilder> requestBuilder;
   std::shared_ptr<cly::RequestModule> requestModule;
+  std::shared_ptr<cly::StorageModuleBase> storageModule;
   std::shared_ptr<std::mutex> mutex = std::make_shared<std::mutex>();
 
   bool is_queue_being_processed = false;

diff --git a/include/countly/constants.hpp b/include/countly/constants.hpp
@@ -1,19 +1,19 @@
 #ifndef COUNTLY_CONSTANTS_HPP_
 #define COUNTLY_CONSTANTS_HPP_
 
+#include "nlohmann/json.hpp"
 #include <cassert>
 #include <chrono>
+#include <climits>
 #include <functional>
 #include <map>
 #include <memory>
 #include <random>
 #include <sstream>
 #include <string>
-#include <climits>
-#include "nlohmann/json.hpp"
 
 #define COUNTLY_SDK_NAME "cpp-native-unknown"
-#define COUNTLY_SDK_VERSION "22.06.4"
+#define COUNTLY_SDK_VERSION "22.09.0"
 #define COUNTLY_POST_THRESHOLD 2000
 #define COUNTLY_KEEPALIVE_INTERVAL 3000
 #define COUNTLY_MAX_EVENTS_DEFAULT 200
@@ -31,7 +31,7 @@ const std::default_random_engine generator(std::chrono::system_clock::now().time
 const std::uniform_int_distribution<int> distribution(1, INT_MAX);
 
 /**
- * Convert map into a string.
+ * Formats the given arguments into a string buffer.
  *
  * @param format formatting string
  * @param args arguments to be formatted
@@ -49,7 +49,7 @@ template <typename... Args> static std::string format_string(const std::string &
 }
 
 /**
- * Convert map into a string.
+ * Gives a string representation of the size of a map.
  *
  * @param m a map containing key-value pairs
  * @return a string object holding size of map.

diff --git a/include/countly/countly_configuration.hpp b/include/countly/countly_configuration.hpp
@@ -27,6 +27,13 @@ struct CountlyConfiguration {
    */
   std::string salt;
 
+  /**
+   * Path to the database.
+   */
+#ifdef COUNTLY_USE_SQLITE
+  std::string databasePath;
+#endif
+
   /**
    * Sets the interval for the automatic update calls
    * min value 1 (1 second), max value 600 (10 minutes)

diff --git a/include/countly/request_module.hpp b/include/countly/request_module.hpp
@@ -8,13 +8,14 @@
 #include "countly/countly_configuration.hpp"
 #include "countly/logger_module.hpp"
 #include "countly/request_builder.hpp"
+#include "countly/storage_module_base.hpp"
 
 namespace cly {
 class RequestModule {
 
 public:
   ~RequestModule();
-  RequestModule(std::shared_ptr<CountlyConfiguration> config, std::shared_ptr<LoggerModule> logger, std::shared_ptr<RequestBuilder> requestBuilder);
+  RequestModule(std::shared_ptr<CountlyConfiguration> config, std::shared_ptr<LoggerModule> logger, std::shared_ptr<RequestBuilder> requestBuilder, std::shared_ptr<StorageModuleBase> storageModule);
 
   HTTPResponse sendHTTP(std::string path, std::string data);
 

diff --git a/include/countly/storage_module_base.hpp b/include/countly/storage_module_base.hpp
@@ -0,0 +1,101 @@
+#ifndef STORAGE_MODULE_BASE_HPP_
+#define STORAGE_MODULE_BASE_HPP_
+#include "countly/countly_configuration.hpp"
+#include "countly/logger_module.hpp"
+#include <string>
+#include <vector>
+
+namespace cly {
+
+class DataEntry {
+private:
+  long long _id;
+  std::string _data;
+
+public:
+  DataEntry(const long long id, const std::string &data) {
+    this->_id = id;
+    this->_data = data;
+  }
+
+  ~DataEntry() {}
+
+  /**
+   * @return id of data entry
+   */
+  long long getId() const { return _id; }
+
+  /**
+   * @return content of data entry
+   */
+  const std::string &getData() const { return _data; }
+};
+
+class StorageModuleBase {
+protected:
+  bool _is_initialized = false;
+  std::shared_ptr<CountlyConfiguration> _configuration;
+  std::shared_ptr<LoggerModule> _logger;
+
+public:
+  StorageModuleBase(std::shared_ptr<CountlyConfiguration> config, std::shared_ptr<LoggerModule> logger) {
+    this->_configuration = config;
+    this->_logger = logger;
+  }
+
+  bool isInitialized() { return _is_initialized; }
+
+  virtual ~StorageModuleBase() {
+    _logger.reset();
+    _configuration.reset();
+  }
+
+  /**
+   * Initialize the storage module.
+   */
+  virtual void init() = 0;
+
+  /**
+   * Returns the count of stored requests.
+   * @return count of stored requests
+   */
+  virtual long long RQCount() = 0;
+
+  /**
+   * Delete all stored requests.
+   */
+  virtual void RQClearAll() = 0;
+
+  /**
+   * Remove front request from the request queue.
+   */
+  virtual void RQRemoveFront() = 0;
+
+  /**
+   * Retrieve the element at the front of the queue. It does not deletes the element in the queue.
+   * @return front request of the queue.
+   */
+  const virtual std::shared_ptr<DataEntry> RQPeekFront() = 0;
+
+  /**
+   * Retrieve all requests in the request queue without removing them.
+   * @return a vector of requests.
+   */
+  virtual std::vector<std::shared_ptr<DataEntry>> RQPeekAll() = 0;
+
+  /**
+   * Remove the front request from the request queue if provided request's id and front request's id do match.
+   * @param request: a shared pointer to front request
+   */
+  virtual void RQRemoveFront(std::shared_ptr<DataEntry> request) = 0;
+
+  /**
+   * Insert element into the request queue at the end.
+   * @param request: content of the request
+   */
+  virtual void RQInsertAtEnd(const std::string &request) = 0;
+};
+
+} // namespace cly
+
+#endif