Implement Easy Logging using a config file #747

sfc-gh-ext-simba-nl · 2024-10-02T19:37:18Z

For SNOW-981602: Implement Easy Logging using a config file

Add several new global attributes:
SF_GLOBAL_CLIENT_CONFIG_FILE: To set the client config file so libsnowflakeclient knows where to look for the config file
SF_GLOBAL_LOG_LEVEL: To allow the user to get the log level since libsnowflakeclient handles log setup
SF_GLOBAL_LOG_PATH: To allow the user to get the log path

include/snowflake/ClientConfigParser.hpp

…file

sfc-gh-ext-simba-hx · 2024-10-11T19:59:55Z

tests/test_unit_logger.c

@@ -25,6 +28,47 @@ void test_log_str_to_level(void **unused) {
 }

 #ifndef _WIN32
+/**


Please add a test case to cover the behavior in log_init(), e.g. having a config file in place, after calling snowflake_global_init() check the log settings are as expected.

sfc-gh-ext-simba-hx · 2024-10-11T20:01:53Z

cpp/lib/ClientConfigParser.cpp

+    {
+#if defined(WIN32) || defined(_WIN64)
+      // 4. Try user home dir
+      std::string homeDir = sf_getenv_s("USERPROFILE", envbuf, sizeof(envbuf));


sf_getenv_s could return NULL and cause runtime exception thrown.

sfc-gh-ext-simba-hx · 2024-10-11T20:06:22Z

cpp/lib/ClientConfigParser.cpp

+
+// Private =====================================================================
+////////////////////////////////////////////////////////////////////////////////
+bool ClientConfigParser::checkFileExists(const std::string& in_filePath)


We are on c++17 already maybe we could use std::filesystem::is_regular_file() directly?

std::filesystem isn't available on osx10.14 and boost has a debug assertion on 32-bit windows with MT build

Can we resolve debug assertion issue with 32-bit windows build and use boost filesystem

I think using boost filesystem was able to fix the issue

sfc-gh-ext-simba-hx · 2024-10-11T20:19:33Z

lib/client.c

+    char client_config_file[MAX_PATH] = {0};
+    snowflake_global_get_attribute(
+      SF_GLOBAL_CLIENT_CONFIG_FILE, client_config_file, sizeof(client_config_file));
+    client_config clientConfig = { .logLevel = "", .logPath = "" };


memory leak currently with the buffer allocated in load_client_config().
When you try to free the buffer, initialize with const string could be a problem.

I think if client config is not initialized .logLevel and .logPath should be null

sfc-gh-ext-simba-hx · 2024-10-11T20:21:58Z

cpp/lib/ClientConfigParser.cpp

+}
+
+////////////////////////////////////////////////////////////////////////////////
+void load_client_config(


Seems we are throwing exception from loadClientConfig(). We need to catch that and return error. C interface doesn't expect exception.
Please add error test cases as well this will be caught if we have error test cases.

sfc-gh-ext-simba-hx · 2024-10-11T20:49:27Z

lib/client.c

@@ -36,6 +37,8 @@ sf_bool DEBUG;
 sf_bool SF_OCSP_CHECK;
 char *SF_HEADER_USER_AGENT = NULL;

+static char *CLIENT_CONFIG_FILE = NULL;
+static char* LOG_LEVEL;


why not to set NULL as others? Better not to add LOG_LEVEL as we've had number loglevel maintained with log_set_level() and log_get_level() already. Having another one could cause mismatch.

sfc-gh-ext-simba-hx · 2024-10-11T20:54:23Z

lib/client.c

+        case SF_GLOBAL_CLIENT_CONFIG_FILE:
+            alloc_buffer_and_copy(&CLIENT_CONFIG_FILE, value);
+            break;
+        case SF_GLOBAL_LOG_LEVEL:


I would prefer SF_GLOBAL_LOG_PATH and SF_GLOBAL_LOG_LEVEL to be read only attributes since application already have enough ways to do log settings.

sfc-gh-ext-simba-hx · 2024-10-11T21:13:07Z

include/snowflake/client.h

-    SF_GLOBAL_OCSP_CHECK
+    SF_GLOBAL_OCSP_CHECK,
+    SF_GLOBAL_CLIENT_CONFIG_FILE,
+    SF_GLOBAL_LOG_LEVEL,


remind me why we need SF_GLOBAL_LOG_LEVEL and SF_GLOBAL_LOG_PATH?
Please having a brief note of the interface and usage in either the ticket or the PR description. That could help understanding the implementation and later for documentation as well.

Updated description

cpp/lib/ClientConfigParser.cpp

sfc-gh-jszczerbinski · 2024-10-14T08:04:41Z

cpp/lib/ClientConfigParser.cpp

+  FILE* configFile;
+  try 
+  {
+    configFile = sf_fopen(&configFile, in_filePath.c_str(), "r");


Use std::fstream and std::filesystem APIs?

Unfortunately std::filesystem isn't in osx10.14 yet so it can't be used and there seems to be a 32-bit windows debug assertion issue with reading file with fstream when built with MT

That means we cannot use std::fstream at all? Can you show me the assertion?

Not exactly, it only happens for Windows 32-bit built with MT. All other configurations seem to work fine. If you prefer, I can ifdef it to not run when it's windows debug and it should work fine. The assertion says something like this:
Debug Assertion Failed!

File: minkernel\crts\ucrt\src\appcrt\lowio\read.cpp

Expression: _osfile(fh) & FOPEN

Is this local issue? Does it appear on CI? I think you might be building the project incorrectly. See: https://stackoverflow.com/questions/5984144/assertion-error-in-crt-calling-osfile-in-vs-2008/6010756#6010756

I am using fstream and boost filesystem on my PR without issues

I'm still running into this issue with the tests run on jenkins. Basically the test run just hangs after it hits the assertion error. In the mean time, I've added an ifdef to to run easy logging when in 32-bit debug. I've locally tested running on /MDd and it runs fine, but currently I think we build with dynamic runtime set to OFF

cpp/lib/ClientConfigParser.cpp

…boost 32-bit windows debug issues

…s://github.com/snowflakedb/libsnowflakeclient into SNOW-981602-implement-easy-logging-config-file

sfc-gh-jszczerbinski · 2024-11-08T09:54:15Z

cpp/lib/ClientConfigParser.cpp

+    ClientConfigParser configParser;
+    configParser.loadClientConfig(in_configFilePath, *out_clientConfig);
+  } catch (std::exception e) {
+    sf_fprintf(stderr, e.what());


Why not use logger here?

sfc-gh-jszczerbinski · 2024-11-08T10:00:09Z

cpp/lib/ClientConfigParser.cpp

+{
+  char envbuf[MAX_PATH + 1];
+  std::string derivedConfigPath = "";
+  if (!in_configFilePath.empty())


Please use ealy return instead of else nesting

sfc-gh-jszczerbinski · 2024-11-08T10:05:28Z

cpp/lib/ClientConfigParser.cpp

+        // USERPROFILE is empty, try HOMEDRIVE and HOMEPATH
+        char* homeDriveEnv;
+        char* homePathEnv;
+        if ((homeDriveEnv = sf_getenv_s("HOMEDRIVE", envbuf, sizeof(envbuf))) && (strlen(homeDriveEnv) != 0) &&


Maybe we can write it like this?

char* homeDriveEnv = sf_getenv_s("HOMEDRIVE", envbuf, sizeof(envbuf) if (homeDriveEnv && strlen(homeDriveEnv) != 0)

sfc-gh-jszczerbinski · 2024-11-08T10:12:05Z

cpp/lib/ClientConfigParser.cpp

+    }
+    else
+    {
+#if defined(WIN32) || defined(_WIN64)


Please move finding home directory to another function.

sfc-gh-jszczerbinski · 2024-11-08T10:24:30Z

cpp/lib/ClientConfigParser.cpp

+  FILE* configFile;
+  try 
+  {
+    configFile = sf_fopen(&configFile, in_filePath.c_str(), "r");


Is this local issue? Does it appear on CI? I think you might be building the project incorrectly. See: https://stackoverflow.com/questions/5984144/assertion-error-in-crt-calling-osfile-in-vs-2008/6010756#6010756

sfc-gh-jszczerbinski · 2024-11-08T10:37:14Z

cpp/lib/ClientConfigParser.cpp

+
+// Private =====================================================================
+////////////////////////////////////////////////////////////////////////////////
+bool ClientConfigParser::checkFileExists(const std::string& in_filePath)


Can we resolve debug assertion issue with 32-bit windows build and use boost filesystem

sfc-gh-jszczerbinski · 2024-11-08T10:44:00Z

cpp/lib/ClientConfigParser.cpp

+
+// Public ======================================================================
+////////////////////////////////////////////////////////////////////////////////
+ClientConfigParser::ClientConfigParser()


Can we change the name? Maybe EasyLoggingConfigParser

sfc-gh-jszczerbinski · 2024-11-08T10:57:19Z

lib/client.c

+    snowflake_global_get_attribute(
+      SF_GLOBAL_CLIENT_CONFIG_FILE, client_config_file, sizeof(client_config_file));
+    client_config clientConfig = { .logLevel = "", .logPath = "" };
+    load_client_config(client_config_file, &clientConfig);


C strings are allocated by this function, we should have corresponding fuction to deallocate the string free_client_config

Right now we are leaking memory

sfc-gh-jszczerbinski · 2024-11-08T10:58:53Z

lib/client.c

+    char client_config_file[MAX_PATH] = {0};
+    snowflake_global_get_attribute(
+      SF_GLOBAL_CLIENT_CONFIG_FILE, client_config_file, sizeof(client_config_file));
+    client_config clientConfig = { .logLevel = "", .logPath = "" };


I think if client config is not initialized .logLevel and .logPath should be null

sfc-gh-jszczerbinski · 2024-11-08T11:02:47Z

include/snowflake/ClientConfigParser.hpp

+{
+namespace Client
+{
+  struct ClientConfigException : public std::exception


Why do we need a class if it has no member variables?

Just using it as a wrapper for exception to pass along the error message. I saw other classes having a similar exception struct, but I can just make the function return false instead of throwing an exception if you prefer.

…file

sfc-gh-ext-simba-nl added 14 commits October 2, 2024 12:35

Implement easy logging

7ab80fb

Add missing function definition

836a82a

Fix linux build errors

c1878f3

Fix conversion issue on linux

21673d4

Fix mac build error and fix logging test failure

806fb9c

Fix mac build error in test case

4a5ad67

Fix setting of global log path

4e05bc7

Use sf_memcpy instead

a79370d

Remove the use of boost algorithm string for compare

be865c1

Fix test issue

f90848d

Fix build issue with test

3823568

Change ifstream to FILE due to 32bit debug read issue

a863b05

Fix linux build issue

994228b

Fix test issues

70bdc60

sfc-gh-ext-simba-hx reviewed Oct 10, 2024

View reviewed changes

include/snowflake/ClientConfigParser.hpp Outdated Show resolved Hide resolved

Rename ifndef for headers to be more consistent

309d118

sfc-gh-ext-simba-nl marked this pull request as ready for review October 11, 2024 16:43

sfc-gh-ext-simba-nl requested a review from a team as a code owner October 11, 2024 16:43

Merge branch 'master' into SNOW-981602-implement-easy-logging-config-…

65e29f1

…file

sfc-gh-ext-simba-hx reviewed Oct 11, 2024

View reviewed changes

sfc-gh-jszczerbinski requested changes Oct 14, 2024

View reviewed changes

sfc-gh-ext-simba-nl added 2 commits October 16, 2024 15:57

Replace boost with std::filesystem, fix possible NULL issues, add tests

21a3b11

Add client_config_parser to include dir

8d97f3a

sfc-gh-ext-simba-nl and others added 12 commits October 28, 2024 17:02

Replace boost with std::filesystem, fix possible NULL issues, add tests

e16a67c

Add client_config_parser to include dir

05331fb

Fix compilation error

f370f9b

Fix return value error

d0cbc0c

Use experimental filesystem for non-windows

e020429

Add stdc++fs to link libraries for unix builds

d351866

Fix macos filesystem error

5cc39ae

Revert from using std::filesystem, add back check file exists due to …

75eb366

…boost 32-bit windows debug issues

SNOW-1744756 add picojson to libsnowflakeclient (#755)

8651788

Fix boost error

48d562f

Fix test cases

79128b4

Fix logger unit tests

98c4924

sfc-gh-ext-simba-nl changed the base branch from master to master-2.0.0 October 29, 2024 00:03

sfc-gh-ext-simba-nl added 4 commits October 28, 2024 17:03

Merge branch 'SNOW-981602-implement-easy-logging-config-file' of http…

9d54bab

…s://github.com/snowflakedb/libsnowflakeclient into SNOW-981602-implement-easy-logging-config-file

Update log level and path precedence logic, add SF_LOG_DEFAULT

a5549aa

Remove accidentally added files

169063c

Fix test case

2c285bf

sfc-gh-jszczerbinski requested changes Nov 8, 2024

View reviewed changes

sfc-gh-ext-simba-nl added 2 commits November 12, 2024 17:51

Merge master

9ede47a

Merge branch 'master' into SNOW-981602-implement-easy-logging-config-…

bc61cd7

…file

sfc-gh-ext-simba-nl changed the base branch from master-2.0.0 to master November 15, 2024 00:52

sfc-gh-ext-simba-nl added 8 commits November 15, 2024 14:57

Use picojson instead of cjson, refactor client config parser

9eaadd0

Add exception to keep track of errors

6c3aa65

Revert accidental inclusions from master-2.0.0 branch

c022028

Add back missing files, fix styling change

b03491a

Disable test case for windows 32-bit debug

0d2e977

Ignore client config loading on win32 debug

36faf07

Add comment explaining the ifdef

be2a7b3

Fix log message

d6a00fe

sfc-gh-ext-simba-nl requested a review from sfc-gh-jszczerbinski November 22, 2024 17:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Easy Logging using a config file #747

Implement Easy Logging using a config file #747

sfc-gh-ext-simba-nl commented Oct 2, 2024 •

edited

Loading

sfc-gh-ext-simba-hx Oct 11, 2024

sfc-gh-ext-simba-hx Oct 11, 2024

sfc-gh-ext-simba-hx Oct 11, 2024 •

edited

Loading

sfc-gh-ext-simba-nl Oct 17, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-ext-simba-nl Nov 21, 2024

sfc-gh-ext-simba-hx Oct 11, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-ext-simba-hx Oct 11, 2024 •

edited

Loading

sfc-gh-ext-simba-hx Oct 11, 2024 •

edited

Loading

sfc-gh-ext-simba-hx Oct 11, 2024

sfc-gh-ext-simba-hx Oct 11, 2024 •

edited

Loading

sfc-gh-ext-simba-nl Oct 15, 2024

sfc-gh-jszczerbinski Oct 14, 2024

sfc-gh-ext-simba-nl Oct 17, 2024 •

edited

Loading

sfc-gh-jszczerbinski Oct 18, 2024

sfc-gh-ext-simba-nl Oct 21, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-ext-simba-nl Nov 21, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-jszczerbinski Nov 8, 2024

sfc-gh-ext-simba-nl Nov 18, 2024

Implement Easy Logging using a config file #747

Are you sure you want to change the base?

Implement Easy Logging using a config file #747

Conversation

sfc-gh-ext-simba-nl commented Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-ext-simba-hx Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-ext-simba-hx Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

sfc-gh-ext-simba-hx Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-ext-simba-hx Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-ext-simba-nl Oct 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-ext-simba-nl commented Oct 2, 2024 •

edited

Loading

sfc-gh-ext-simba-hx Oct 11, 2024 •

edited

Loading

sfc-gh-ext-simba-hx Oct 11, 2024 •

edited

Loading

sfc-gh-ext-simba-hx Oct 11, 2024 •

edited

Loading

sfc-gh-ext-simba-hx Oct 11, 2024 •

edited

Loading

sfc-gh-ext-simba-nl Oct 17, 2024 •

edited

Loading