From 2e77402b38662692953d6992cdf95cfadbd2b357 Mon Sep 17 00:00:00 2001 From: "luke.power" Date: Wed, 10 Jan 2024 13:42:23 +0000 Subject: [PATCH 1/5] Adding Logging Standards based on MBTP Standards. --- docs/standards/logging.md | 149 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 docs/standards/logging.md diff --git a/docs/standards/logging.md b/docs/standards/logging.md new file mode 100644 index 00000000..cd5fd807 --- /dev/null +++ b/docs/standards/logging.md @@ -0,0 +1,149 @@ +--- +layout: standard +order: 1 +title: Logging for applications / systems / services with centralised logging tooling +date: 2024-01-05 # this should be the date that the content was most recently amended or formally reviewed +id: SEGAS-00014 # Set unique ID for standard +# use `tags: []` for no tags +# Check https://ho-cto.github.io/engineering-guidance-and-standards/tags/ for existing tags +# Note: tags must use sentence case capitalisation +tags: + - SRE + - Monitoring + - Observability + - Logging +related: + sections: + - title: Securing application logging + items: + - text: Securing application logging + href: /patterns/securing-application-logging/ +--- + + + +to-do + + + +--- + +## Requirement(s) + + + + + +- [Application / service / system logs must have at least the following fields as a minimum](#Application--service--system-logs-must-have-at-least-the-following-fields-as-a-minimum) +- [Application / service / system logs must be forwarded to the supported log aggregation tooling.](#application--service--system-logs-must-be-forwarded-to-the-supported-log-aggregation-tooling) +- [Application / service / system logs must be rotated daily and no more than 100 MB of logs to be retained on the pods at any given time.](#application--service--system-logs-must-be-rotated-daily-and-no-more-than-100-mb-of-logs-to-be-retained-on-the-pods-at-any-given-time) +- [Application / service / system logs must have an identifiable source (e.g. access / audit / application etc.) in the log aggregation tooling.](#application--service--system-logs-must-have-an-identifiable-source-eg-access--audit--application-etc-in-the-log-aggregation-tooling) +- [Application / service / system log messages must be consistent per log source (e.g. access / audit / application etc)](#application--service--system-log-messages-must-be-consistent-per-log-source-eg-access--audit--application-etc) +- [Application / service / system log messages must have a consistent time source](#application--service--system-log-messages-must-have-a-consistent-time-source) +- [Application / service / system log messages must not reveal any person identifiable information](#application--service--system-log-messages-must-not-reveal-any-person-identifiable-information) +- [Application / service / system log messages must not reveal any sensitive information.](#application--service--system-log-messages-must-not-reveal-any-sensitive-information) +- [Application / service / system log messages should only be used for metrics, dashboarding and alerting as a final option. All efforts should be made to migrate log-based monitoring towards purpose-built monitoring tooling](#application--service--system-log-messages-should-only-be-used-for-metrics-dashboarding-and-alerting-as-a-final-option-all-efforts-should-be-made-to-migrate-log-based-monitoring-towards-purpose-built-monitoring-tooling) +- [Default log level for production environments is INFO level; there must be no DEBUG / TRACE level debug messages in Production environments.](#default-log-level-for-production-environments-is-info-level-there-must-be-no-debug--trace-level-debug-messages-in-production-environments) +- [Given a single [HTTP] request, you can trace its journey from start to finish (within system boundaries). This includes calls to and from upstream / downstream applications / services / systems and message queuing and database interactions.] (#given-a-single-http-request-you-can-trace-its-journey-from-start-to-finish-within-system-boundaries-this-includes-calls-to-and-from-upstream--downstream-applications--services--systems-and-message-queuing-and-database-interactions) +- [Given a single [HTTP] request, you can trace its journey from start to finish measuring time spent on each individual action, including calls to external applications / services / systems.](#given-a-single-http-request-you-can-trace-its-journey-from-start-to-finish-measuring-time-spent-on-each-individual-action-including-calls-to-external-applications--services--systems) +- [Given a single [HTTP] request, the request must return an appropriate HTTP response code in its logs.](#given-a-single-http-request-the-request-must-return-an-appropriate-http-response-code) + + +### Application / service / system logs must have at least the following fields as a minimum + +
INFORMATIONDESCRIPTION
Date / TimeEach log message must include date and time (millisecond accuracy) (UTC).
Application / Service / SystemEach log message must be attributed to an application / service / system (i.e. hostname).
Log levelEach log message must have an appropriate log level (INFO, WARN, ERROR, etc.)
Log messageEach log message must have a log message
+ +### Application / service / system logs must be forwarded to the supported log aggregation tooling. + +Application / service / system logs must be forwarded to the supported log aggregation tooling for centralised and persistent storage, and ease of querying log information across distributed applications and services. + +### Application / service / system logs must be rotated daily and no more than 100 MB of logs to be retained on the pods at any given time. + +Where log files are persisted locally, log files must be rotated to prevent filling up local disk space. Furthermore, no more than 100 MB of logs are to be persisted should your application / service / system generate lots of logs. + +### Application / service / system logs must have an identifiable source (e.g. access / audit / application etc.) in the log aggregation tooling. + +Application / service / systems can have a variety of logs, including application, error, and access logs; so it must be possible to identify the source of the log data in the log aggregation tooling. + +### Application / service / system log messages must be consistent per log source (e.g. access / audit / application etc) + +Log messages must follow a consistent log messaging format per log source to allow for easy processing and translation of logs into other formats. + +### Application / service / system log messages must have a consistent time source + +All applications / services / system logs must have a consistent time source so that log messages can be viewed / sorted. + +### Application / service / system log messages must not reveal any person identifiable information + +Information and data sets that can be used to directly identify a person should not output to logs regardless of log level. + +Examples of such information / data sets include, +* Passport number, Issuing Country, Expiry Dates +* BRP/C number, Date of Birth + +With the examples above, the information by themselves do not mean very much, after all a BRP/C number is just a "random" sequence of characters, but in combination with the date of birth it can be used to uniquely identify an individual. + +Such information should not be stored outside of the controls of the application / service / system that masters it due to regulatory controls (i.e. data persistence, retention, and security controls). + +If you are unsure what is person identifiable information, consult with your security specialists. + + +### Application / service / system log messages must not reveal any sensitive information. + +Information and data sets that contain sensitive information should not be output to logs regardless of log level. + +Such information should not be stored outside of the controls of the application / service / system that masters it due to regulatory controls (i.e. data persistence, retention, and security controls). + +If you are unsure what is sensitive information, consult with your security specialists. + +### Application / service / system log messages should only be used for metrics, dashboarding and alerting as a final option. All efforts should be made to migrate log-based monitoring towards purpose-built monitoring tooling + +Although acceptable in some cases, it is strongly recommended that you do not use information contained in log messages as the primary method by which you report on application health, performance and business metrics. Information in logs can be difficult to parse, aggregate and quite brittle. + +The use of your approved tooling for application / service / system monitoring and alerting should always be prioritised over the use of log-based monitoring. + +### Default log level for production environments is INFO level; there must be no DEBUG / TRACE level debug messages in Production environments. + +Application / service / system log messages should not output DEBUG or TRACE level messages in PRODUCTION. + +DEBUG / TRACE level messages, other than filling up logs, often reveal a lot of information and data that should not be output during standard execution. If it is required, then this is by exception and will require approval from the appropriate system / data owners. + +### Given a single [HTTP] request, you can trace its journey from start to finish (within system boundaries). This includes calls to and from upstream / downstream applications / services / systems and message queuing and database interactions. + +In highly distributed applications / systems / services, particularly with microservice based architectures, it is useful to be able to see all the parts of a single transaction from start to finish, including calls to external systems and services. Being able to do so enables troubleshooting and debugging of complex issues and identifying performance bottlenecks. + +### Given a single [HTTP] request, you can trace its journey from start to finish measuring time spent on each individual action, including calls to external applications / services / systems. + +In highly distributed applications / systems / services, particularly with microservice based architectures, it is useful to be able to see the time spent on each part of a single transaction from start to finish, including calls to external systems and services. Being able to do so enables troubleshooting and debugging of complex issues and identifying performance bottlenecks. + +### Given a single [HTTP] request, the request must return an appropriate HTTP response code. + +A response to a request must include an appropriate HTTP response code, doing so will aide monitoring tools in determining genuine failure / threat scenarios as well as determining application / service / system health. + +
CODEDESCRIPTION
1xxInformational response codes
2xxSuccessful responses
3xxRedirect responses
4xxClient errors
5xxServer errors
+ +--- From 45aa15c3e6f5b2debc1c473ec31755b93e30530e Mon Sep 17 00:00:00 2001 From: "luke.power" Date: Wed, 10 Jan 2024 13:56:07 +0000 Subject: [PATCH 2/5] Added description to MBTP logging standards. --- docs/standards/logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/standards/logging.md b/docs/standards/logging.md index cd5fd807..4c147aac 100644 --- a/docs/standards/logging.md +++ b/docs/standards/logging.md @@ -22,7 +22,7 @@ related: -to-do +Logging tools are one of the primary ways of debugging application problems and identifying the root cause of incidents. However, logs need to be in a standard and useful format so that they are not only useful for the application's original developers, but also support engineers down the line. However, logs are sometimes relied on to heavily for problem detection, over purpose chosen monitoring tooling. Therefore these standards provide guidance on the proper utilsation of application logs, as well as their format. -- [Application / service / system logs must have at least the following fields as a minimum](#Application--service--system-logs-must-have-at-least-the-following-fields-as-a-minimum) -- [Application / service / system logs must be forwarded to the supported log aggregation tooling.](#application--service--system-logs-must-be-forwarded-to-the-supported-log-aggregation-tooling) -- [Application / service / system logs must be rotated daily and no more than 100 MB of logs to be retained on the pods at any given time.](#application--service--system-logs-must-be-rotated-daily-and-no-more-than-100-mb-of-logs-to-be-retained-on-the-pods-at-any-given-time) -- [Application / service / system logs must have an identifiable source (e.g. access / audit / application etc.) in the log aggregation tooling.](#application--service--system-logs-must-have-an-identifiable-source-eg-access--audit--application-etc-in-the-log-aggregation-tooling) -- [Application / service / system log messages must be consistent per log source (e.g. access / audit / application etc)](#application--service--system-log-messages-must-be-consistent-per-log-source-eg-access--audit--application-etc) -- [Application / service / system log messages must have a consistent time source](#application--service--system-log-messages-must-have-a-consistent-time-source) -- [Application / service / system log messages must not reveal any person identifiable information](#application--service--system-log-messages-must-not-reveal-any-person-identifiable-information) -- [Application / service / system log messages must not reveal any sensitive information.](#application--service--system-log-messages-must-not-reveal-any-sensitive-information) -- [Application / service / system log messages should only be used for metrics, dashboarding and alerting as a final option. All efforts should be made to migrate log-based monitoring towards purpose-built monitoring tooling](#application--service--system-log-messages-should-only-be-used-for-metrics-dashboarding-and-alerting-as-a-final-option-all-efforts-should-be-made-to-migrate-log-based-monitoring-towards-purpose-built-monitoring-tooling) -- [Default log level for production environments is INFO level; there must be no DEBUG / TRACE level debug messages in Production environments.](#default-log-level-for-production-environments-is-info-level-there-must-be-no-debug--trace-level-debug-messages-in-production-environments) -- [Given a single [HTTP] request, you can trace its journey from start to finish (within system boundaries). This includes calls to and from upstream / downstream applications / services / systems and message queuing and database interactions.] (#given-a-single-http-request-you-can-trace-its-journey-from-start-to-finish-within-system-boundaries-this-includes-calls-to-and-from-upstream--downstream-applications--services--systems-and-message-queuing-and-database-interactions) -- [Given a single [HTTP] request, you can trace its journey from start to finish measuring time spent on each individual action, including calls to external applications / services / systems.](#given-a-single-http-request-you-can-trace-its-journey-from-start-to-finish-measuring-time-spent-on-each-individual-action-including-calls-to-external-applications--services--systems) -- [Given a single [HTTP] request, the request must return an appropriate HTTP response code in its logs.](#given-a-single-http-request-the-request-must-return-an-appropriate-http-response-code) +- [Service logs must have the minimum required field](#service-logs-must-have-the-minimum-required-fields) +- [Service logs must be forwarded to log aggregators](#service-logs-must-be-forwarded-to-log-aggregators) +- [Service logs must be rotated daily and no more than 100 MB of logs to be retained](#service-logs-must-be-rotated-daily-and-no-more-than-100-mb-of-logs-to-be-retained) +- [ervice logs must have an identifiable source in the log aggregator](#service-logs-must-have-an-identifiable-source-in-the-log-aggregator) +- [Service log messages must be consistent per log source](#service-log-messages-must-be-consistent-per-log-source) +- [Service log messages must have a consistent time source](#service-log-messages-must-have-a-consistent-time-source) +- [Service log messages must not reveal any person identifiable information](#service-log-messages-must-not-reveal-any-person-identifiable-information) +- [Service log messages must not reveal any sensitive information.](#service-log-messages-must-not-reveal-any-sensitive-information) +- [Service log messages should only be used for metrics, dashboarding and alerting as a last resort](#service-log-messages-should-only-be-used-for-metrics-dashboarding-and-alerting-as-a-last-resort) +- [There must be no DEBUG / TRACE level debug messages in Production environments.](#there-must-be-no-debug--trace-level-debug-messages-in-production-environments) +- [Any HTTP request can be traced from start to finish (within system boundaries)](#any-http-request-can-be-traced-from-start-to-finish-within-system-boundaries) +- [The time any HTTP request spends on each individual action can be measured](#the-time-any-http-request-spends-on-each-individual-action-can-be-measured) +- [Any HTTP request must return an appropriate HTTP response code](#any-http-request-must-return-an-appropriate-http-response-code) +### Service logs must have the minimum required fields -### Application / service / system logs must have at least the following fields as a minimum +
INFORMATIONDESCRIPTION
Date / TimeEach log message must include date and time (millisecond accuracy) (UTC).
ServiceEach log message must be attributed to an service (i.e. hostname).
Log levelEach log message must have an appropriate log level (INFO, WARN, ERROR, etc.)
Log messageEach log message must have a log message
-
INFORMATIONDESCRIPTION
Date / TimeEach log message must include date and time (millisecond accuracy) (UTC).
Application / Service / SystemEach log message must be attributed to an application / service / system (i.e. hostname).
Log levelEach log message must have an appropriate log level (INFO, WARN, ERROR, etc.)
Log messageEach log message must have a log message
+### Service logs must be forwarded to log aggregators -### Application / service / system logs must be forwarded to the supported log aggregation tooling. +Service logs must be forwarded to the supported log aggregation tooling for centralised and persistent storage, and ease of querying log information across distributed applications and services. -Application / service / system logs must be forwarded to the supported log aggregation tooling for centralised and persistent storage, and ease of querying log information across distributed applications and services. +### Service logs must be rotated daily and no more than 100 MB of logs to be retained + -### Application / service / system logs must be rotated daily and no more than 100 MB of logs to be retained on the pods at any given time. +Where log files are persisted locally, log files must be rotated to prevent filling up local disk space. Furthermore, no more than 100 MB of logs are to be persisted should your service generate lots of logs. -Where log files are persisted locally, log files must be rotated to prevent filling up local disk space. Furthermore, no more than 100 MB of logs are to be persisted should your application / service / system generate lots of logs. +### Service logs must have an identifiable source in the log aggregator -### Application / service / system logs must have an identifiable source (e.g. access / audit / application etc.) in the log aggregation tooling. +Services can have a variety of logs, including application, error, and access logs; so it must be possible to identify the source of the log data in the log aggregation tooling. -Application / service / systems can have a variety of logs, including application, error, and access logs; so it must be possible to identify the source of the log data in the log aggregation tooling. +### Service log messages must be consistent per log source -### Application / service / system log messages must be consistent per log source (e.g. access / audit / application etc) +Log messages must follow a consistent log messaging format per log source (e.g. access / audit / application etc) to allow for easy processing and translation of logs into other formats. -Log messages must follow a consistent log messaging format per log source to allow for easy processing and translation of logs into other formats. - -### Application / service / system log messages must have a consistent time source +### Service log messages must have a consistent time source All applications / services / system logs must have a consistent time source so that log messages can be viewed / sorted. -### Application / service / system log messages must not reveal any person identifiable information +### Service log messages must not reveal any person identifiable information + Information and data sets that can be used to directly identify a person should not output to logs regardless of log level. @@ -107,42 +108,43 @@ Examples of such information / data sets include, With the examples above, the information by themselves do not mean very much, after all a BRP/C number is just a "random" sequence of characters, but in combination with the date of birth it can be used to uniquely identify an individual. -Such information should not be stored outside of the controls of the application / service / system that masters it due to regulatory controls (i.e. data persistence, retention, and security controls). +Such information should not be stored outside of the controls of the service that masters it due to regulatory controls (i.e. data persistence, retention, and security controls). If you are unsure what is person identifiable information, consult with your security specialists. - -### Application / service / system log messages must not reveal any sensitive information. +### Service log messages must not reveal any sensitive information + Information and data sets that contain sensitive information should not be output to logs regardless of log level. -Such information should not be stored outside of the controls of the application / service / system that masters it due to regulatory controls (i.e. data persistence, retention, and security controls). +Such information should not be stored outside of the controls of the service that masters it due to regulatory controls (i.e. data persistence, retention, and security controls). If you are unsure what is sensitive information, consult with your security specialists. -### Application / service / system log messages should only be used for metrics, dashboarding and alerting as a final option. All efforts should be made to migrate log-based monitoring towards purpose-built monitoring tooling - +### Service log messages should only be used for metrics, dashboarding and alerting as a last resort Although acceptable in some cases, it is strongly recommended that you do not use information contained in log messages as the primary method by which you report on application health, performance and business metrics. Information in logs can be difficult to parse, aggregate and quite brittle. -The use of your approved tooling for application / service / system monitoring and alerting should always be prioritised over the use of log-based monitoring. +The use of your approved tooling for service monitoring and alerting should always be prioritised over the use of log-based monitoring. -### Default log level for production environments is INFO level; there must be no DEBUG / TRACE level debug messages in Production environments. +### There must be no DEBUG / TRACE level messages in Production environments + -Application / service / system log messages should not output DEBUG or TRACE level messages in PRODUCTION. +Service log messages should not output DEBUG or TRACE level messages in PRODUCTION. The Default log level for production environments is INFO level. DEBUG / TRACE level messages, other than filling up logs, often reveal a lot of information and data that should not be output during standard execution. If it is required, then this is by exception and will require approval from the appropriate system / data owners. -### Given a single [HTTP] request, you can trace its journey from start to finish (within system boundaries). This includes calls to and from upstream / downstream applications / services / systems and message queuing and database interactions. + +### Any HTTP request can be traced from start to finish (within system boundaries) In highly distributed applications / systems / services, particularly with microservice based architectures, it is useful to be able to see all the parts of a single transaction from start to finish, including calls to external systems and services. Being able to do so enables troubleshooting and debugging of complex issues and identifying performance bottlenecks. -### Given a single [HTTP] request, you can trace its journey from start to finish measuring time spent on each individual action, including calls to external applications / services / systems. +### The time any HTTP request spends on each individual action can be measured In highly distributed applications / systems / services, particularly with microservice based architectures, it is useful to be able to see the time spent on each part of a single transaction from start to finish, including calls to external systems and services. Being able to do so enables troubleshooting and debugging of complex issues and identifying performance bottlenecks. -### Given a single [HTTP] request, the request must return an appropriate HTTP response code. +### Any HTTP request must return an appropriate HTTP response code -A response to a request must include an appropriate HTTP response code, doing so will aide monitoring tools in determining genuine failure / threat scenarios as well as determining application / service / system health. +A response to a request must include an appropriate HTTP response code, doing so will aide monitoring tools in determining genuine failure / threat scenarios as well as determining service health.
CODEDESCRIPTION
1xxInformational response codes
2xxSuccessful responses
3xxRedirect responses
4xxClient errors
5xxServer errors
From d803dc3e38dd92309a667ed38e568f8ae92d61f7 Mon Sep 17 00:00:00 2001 From: "luke.power" Date: Wed, 24 Jan 2024 13:21:27 +0000 Subject: [PATCH 4/5] Combining logging standards and reducing duplication with the existing logging patterns. --- docs/patterns/securing-application-logging.md | 12 ++++ docs/standards/logging.md | 59 +++---------------- 2 files changed, 19 insertions(+), 52 deletions(-) diff --git a/docs/patterns/securing-application-logging.md b/docs/patterns/securing-application-logging.md index 3a01d4a4..72549e4f 100644 --- a/docs/patterns/securing-application-logging.md +++ b/docs/patterns/securing-application-logging.md @@ -65,6 +65,18 @@ Logging full payloads should be avoided. However, logging full payloads can be u ## Considerations +### Consider how information can combine to form PII + +Examples of PII include: +* Passport number, Issuing Country, Expiry Dates +* BRP/C number, Date of Birth + +With the examples above, the information by themselves do not mean very much, after all a BRP/C number is just a "random" sequence of characters; but in combination with the date of birth it can be used to uniquely identify an individual. + +If you are unsure what is person identifiable information, consult with your security specialists. + +Information and data sets that contain sensitive information should not be output to logs regardless of log level. + ### Consider whether logging statements are necessary Sometimes logging is used for fast-feedback loops when prototyping application functionality, but eventually become redundant, or the application they provide does not provide additional operational or debugging value for an operator searching through logs. diff --git a/docs/standards/logging.md b/docs/standards/logging.md index 4c8ad6cd..303e38c8 100644 --- a/docs/standards/logging.md +++ b/docs/standards/logging.md @@ -1,9 +1,9 @@ --- layout: standard order: 1 -title: Logging for applications / systems / services with centralised logging tooling +title: Logging for services with centralised logging aggregation date: 2024-01-05 # this should be the date that the content was most recently amended or formally reviewed -id: SEGAS-00014 # Set unique ID for standard +id: SEGAS-00015 # Set unique ID for standard # use `tags: []` for no tags # Check https://ho-cto.github.io/engineering-guidance-and-standards/tags/ for existing tags # Note: tags must use sentence case capitalisation @@ -76,12 +76,13 @@ Use HTML URL encoding as in the 'Notes on links' above, to ensure that links to
INFORMATIONDESCRIPTION
Date / TimeEach log message must include date and time (millisecond accuracy) (UTC).
ServiceEach log message must be attributed to an service (i.e. hostname).
Log levelEach log message must have an appropriate log level (INFO, WARN, ERROR, etc.)
Log messageEach log message must have a log message
+The time source for all service logs must be consistent so that messages can be viewed & sorted easily. + ### Service logs must be forwarded to log aggregators Service logs must be forwarded to the supported log aggregation tooling for centralised and persistent storage, and ease of querying log information across distributed applications and services. ### Service logs must be rotated daily and no more than 100 MB of logs to be retained - Where log files are persisted locally, log files must be rotated to prevent filling up local disk space. Furthermore, no more than 100 MB of logs are to be persisted should your service generate lots of logs. @@ -89,63 +90,17 @@ Where log files are persisted locally, log files must be rotated to prevent fill Services can have a variety of logs, including application, error, and access logs; so it must be possible to identify the source of the log data in the log aggregation tooling. -### Service log messages must be consistent per log source - -Log messages must follow a consistent log messaging format per log source (e.g. access / audit / application etc) to allow for easy processing and translation of logs into other formats. - -### Service log messages must have a consistent time source - -All applications / services / system logs must have a consistent time source so that log messages can be viewed / sorted. - -### Service log messages must not reveal any person identifiable information - - -Information and data sets that can be used to directly identify a person should not output to logs regardless of log level. - -Examples of such information / data sets include, -* Passport number, Issuing Country, Expiry Dates -* BRP/C number, Date of Birth - -With the examples above, the information by themselves do not mean very much, after all a BRP/C number is just a "random" sequence of characters, but in combination with the date of birth it can be used to uniquely identify an individual. - -Such information should not be stored outside of the controls of the service that masters it due to regulatory controls (i.e. data persistence, retention, and security controls). - -If you are unsure what is person identifiable information, consult with your security specialists. - -### Service log messages must not reveal any sensitive information - - -Information and data sets that contain sensitive information should not be output to logs regardless of log level. - -Such information should not be stored outside of the controls of the service that masters it due to regulatory controls (i.e. data persistence, retention, and security controls). - -If you are unsure what is sensitive information, consult with your security specialists. - ### Service log messages should only be used for metrics, dashboarding and alerting as a last resort Although acceptable in some cases, it is strongly recommended that you do not use information contained in log messages as the primary method by which you report on application health, performance and business metrics. Information in logs can be difficult to parse, aggregate and quite brittle. The use of your approved tooling for service monitoring and alerting should always be prioritised over the use of log-based monitoring. ### There must be no DEBUG / TRACE level messages in Production environments - - -Service log messages should not output DEBUG or TRACE level messages in PRODUCTION. The Default log level for production environments is INFO level. - -DEBUG / TRACE level messages, other than filling up logs, often reveal a lot of information and data that should not be output during standard execution. If it is required, then this is by exception and will require approval from the appropriate system / data owners. - - -### Any HTTP request can be traced from start to finish (within system boundaries) - -In highly distributed applications / systems / services, particularly with microservice based architectures, it is useful to be able to see all the parts of a single transaction from start to finish, including calls to external systems and services. Being able to do so enables troubleshooting and debugging of complex issues and identifying performance bottlenecks. - -### The time any HTTP request spends on each individual action can be measured - -In highly distributed applications / systems / services, particularly with microservice based architectures, it is useful to be able to see the time spent on each part of a single transaction from start to finish, including calls to external systems and services. Being able to do so enables troubleshooting and debugging of complex issues and identifying performance bottlenecks. -### Any HTTP request must return an appropriate HTTP response code +Refer to our [managing secrets]({{ '/patterns/securing-application-logging/' | url }}) pattern for more details. -A response to a request must include an appropriate HTTP response code, doing so will aide monitoring tools in determining genuine failure / threat scenarios as well as determining service health. +### Service log messages must not reveal any sensitive information or person identifiable information -
CODEDESCRIPTION
1xxInformational response codes
2xxSuccessful responses
3xxRedirect responses
4xxClient errors
5xxServer errors
+Refer to our [managing secrets]({{ '/patterns/securing-application-logging/' | url }}) pattern for more details. --- From 8a29496d9b1c016ea336cfbf70d202399923a996 Mon Sep 17 00:00:00 2001 From: "luke.power" Date: Wed, 24 Jan 2024 17:46:33 +0000 Subject: [PATCH 5/5] Fixing link errors and combining more standards. --- docs/patterns/securing-application-logging.md | 12 +-- docs/standards/logging.md | 89 ++++++------------- 2 files changed, 34 insertions(+), 67 deletions(-) diff --git a/docs/patterns/securing-application-logging.md b/docs/patterns/securing-application-logging.md index 72549e4f..93b144b9 100644 --- a/docs/patterns/securing-application-logging.md +++ b/docs/patterns/securing-application-logging.md @@ -19,7 +19,7 @@ related: --- It is common to log user information from an application to assist operations or debugging. Some middleware might be configured to log full configuration, data, or request/response objects, which may contain Personally Identifiable Information (PII), authentication tokens or application secrets. - + Logging any of these can make information available to people that have access to log aggregation platforms, or log storage, but are not legally allowed to view PII, or should not otherwise have access to that information. User authentication tokens and application secrets can be used to impersonate users and elevate privilege. [Sensitive Data Exposure](https://owasp.org/www-project-top-ten/2017/A3_2017-Sensitive_Data_Exposure) has been a recognised risk to secure application development historically. Our [managing secrets]({{ '/standards/managing-secrets/' | url }}) standard has requirements on ensuring logs do not contain tokens or other secrets. @@ -67,14 +67,14 @@ Logging full payloads should be avoided. However, logging full payloads can be u ### Consider how information can combine to form PII -Examples of PII include: +Examples of combined PII include: * Passport number, Issuing Country, Expiry Dates * BRP/C number, Date of Birth - -With the examples above, the information by themselves do not mean very much, after all a BRP/C number is just a "random" sequence of characters; but in combination with the date of birth it can be used to uniquely identify an individual. - + +With the examples above, the information by themselves do not mean very much. After all, a BRP/C number is just a "random" sequence of characters; but in combination with the date of birth it can be used to uniquely identify an individual. + If you are unsure what is person identifiable information, consult with your security specialists. - + Information and data sets that contain sensitive information should not be output to logs regardless of log level. ### Consider whether logging statements are necessary diff --git a/docs/standards/logging.md b/docs/standards/logging.md index 303e38c8..400b8535 100644 --- a/docs/standards/logging.md +++ b/docs/standards/logging.md @@ -2,11 +2,8 @@ layout: standard order: 1 title: Logging for services with centralised logging aggregation -date: 2024-01-05 # this should be the date that the content was most recently amended or formally reviewed -id: SEGAS-00015 # Set unique ID for standard -# use `tags: []` for no tags -# Check https://ho-cto.github.io/engineering-guidance-and-standards/tags/ for existing tags -# Note: tags must use sentence case capitalisation +date: 2024-01-24 +id: SEGAS-00015 tags: - SRE - Monitoring @@ -20,87 +17,57 @@ related: href: /patterns/securing-application-logging/ --- - +Logging tools are one of the primary tools to debug application problems and identifying the root cause of incidents. However, logs need to be formatted in a standard way so that they are not only useful for the application's original developers, but also those supporting the application later in its life-cycle. -Logging tools are one of the primary ways of debugging application problems and identifying the root cause of incidents. However, logs need to be in a standard and useful format so that they are not only useful for the application's original developers, but also support engineers down the line. However, logs are sometimes relied on to heavily for problem detection, over purpose chosen monitoring tooling. Therefore these standards provide guidance on the proper utilsation of application logs, as well as their format. +However, logs are sometimes relied on too heavily for problem detection, when this role would be better suited to purpose chosen monitoring tooling. - +Therefore these standards provide guidance on both the proper format of application logs, as well as their proper usage. --- -## Requirement(s) - - +## Requirements - - -- [Service logs must have the minimum required field](#service-logs-must-have-the-minimum-required-fields) -- [Service logs must be forwarded to log aggregators](#service-logs-must-be-forwarded-to-log-aggregators) -- [Service logs must be rotated daily and no more than 100 MB of logs to be retained](#service-logs-must-be-rotated-daily-and-no-more-than-100-mb-of-logs-to-be-retained) -- [ervice logs must have an identifiable source in the log aggregator](#service-logs-must-have-an-identifiable-source-in-the-log-aggregator) -- [Service log messages must be consistent per log source](#service-log-messages-must-be-consistent-per-log-source) -- [Service log messages must have a consistent time source](#service-log-messages-must-have-a-consistent-time-source) -- [Service log messages must not reveal any person identifiable information](#service-log-messages-must-not-reveal-any-person-identifiable-information) -- [Service log messages must not reveal any sensitive information.](#service-log-messages-must-not-reveal-any-sensitive-information) -- [Service log messages should only be used for metrics, dashboarding and alerting as a last resort](#service-log-messages-should-only-be-used-for-metrics-dashboarding-and-alerting-as-a-last-resort) -- [There must be no DEBUG / TRACE level debug messages in Production environments.](#there-must-be-no-debug--trace-level-debug-messages-in-production-environments) -- [Any HTTP request can be traced from start to finish (within system boundaries)](#any-http-request-can-be-traced-from-start-to-finish-within-system-boundaries) -- [The time any HTTP request spends on each individual action can be measured](#the-time-any-http-request-spends-on-each-individual-action-can-be-measured) -- [Any HTTP request must return an appropriate HTTP response code](#any-http-request-must-return-an-appropriate-http-response-code) - -### Service logs must have the minimum required fields - -
INFORMATIONDESCRIPTION
Date / TimeEach log message must include date and time (millisecond accuracy) (UTC).
ServiceEach log message must be attributed to an service (i.e. hostname).
Log levelEach log message must have an appropriate log level (INFO, WARN, ERROR, etc.)
Log messageEach log message must have a log message
- -The time source for all service logs must be consistent so that messages can be viewed & sorted easily. - -### Service logs must be forwarded to log aggregators +### Service logs MUST be forwarded to log aggregators Service logs must be forwarded to the supported log aggregation tooling for centralised and persistent storage, and ease of querying log information across distributed applications and services. -### Service logs must be rotated daily and no more than 100 MB of logs to be retained +### Service logs MUST be rotated daily and no more than 100 MB of logs to be retained Where log files are persisted locally, log files must be rotated to prevent filling up local disk space. Furthermore, no more than 100 MB of logs are to be persisted should your service generate lots of logs. -### Service logs must have an identifiable source in the log aggregator +### Service logs MUST have an identifiable source in the log aggregator Services can have a variety of logs, including application, error, and access logs; so it must be possible to identify the source of the log data in the log aggregation tooling. ### Service log messages should only be used for metrics, dashboarding and alerting as a last resort Although acceptable in some cases, it is strongly recommended that you do not use information contained in log messages as the primary method by which you report on application health, performance and business metrics. Information in logs can be difficult to parse, aggregate and quite brittle. - + The use of your approved tooling for service monitoring and alerting should always be prioritised over the use of log-based monitoring. -### There must be no DEBUG / TRACE level messages in Production environments +### There MUST be no DEBUG / TRACE level messages in Production environments -Refer to our [managing secrets]({{ '/patterns/securing-application-logging/' | url }}) pattern for more details. +Refer to our [securing application logging]({{ '/patterns/securing-application-logging/#take-care-when-using-detailed-logging-to-support-application-debugging' | url }}) pattern for more details. -### Service log messages must not reveal any sensitive information or person identifiable information +### Service log messages MUST not reveal any sensitive information or person identifiable information -Refer to our [managing secrets]({{ '/patterns/securing-application-logging/' | url }}) pattern for more details. +Refer to our [securing application logging]({{ '/patterns/securing-application-logging/#consider-how-information-can-combine-to-form-pii' | url }}) pattern for more details. ---