SLIC: Rank count and list formatting

[gberg617]

I'm trying to write to a log file using both GenericOutputStream (for messages from the root rank), and LumberjackStream (for messages from another rank that might need to be filtered). Traditionally, the formatting of this log file would look something like this:

[INFO ][2024/09/20 18:18:57][ 0][ 0][0.000000e+00] Message from root rank
[INFO ][2024/09/20 18:18:57][ 2][ 0][2.200000e-02] Message from rank 2

The number in the third pair of square brackets above indicates the rank that creates the message. Messages that are filtered typically print only the first rank.

There are two issues I'm encountering trying to format this:

1. GenericOutputStream does not have a rank associated with it. The formatting of the <RANK> keyword will just be empty. For all of my applications, I prefer the rank to always print "0", and rank count to be "1" so that it has consistent formatting. Is there a way to override this formatting? Or would you recommend that I create my own stream class to do this?
2. Lumberjack streams now have the trailing dots "..." for the rank list, even if the rank count is 1. Is it possible at all to disable the dots so that I just get the first number in the rank list? This would also help with consistency in formatting.

Any guidance on this would be greatly appreciated!

[gberg617]

@white238 @bmhan12

[bmhan12]

GenericOutputStream does not have a rank associated with it. The formatting of the keyword will just be empty. For all of my applications, I prefer the rank to always print "0", and rank count to be "1" so that it has consistent formatting. Is there a way to override this formatting? Or would you recommend that I create my own stream class to do this?

GenericOutputStream is for non-MPI use cases and builds without MPI, so there is not a way to get the rank information (the "" parameters):

axom/src/axom/slic/streams/GenericOutputStream.cpp

Lines 97 to 103 in 6f5eaa3

	(*m_stream) << this->getFormatedMessage(message::getLevelAsString(msgLevel),
	message,
	tagName,
	"",
	"",
	fileName,
	line);

You should use SynchronizedStream for unfiltered MPI messages. The rank count will always be 1 (no filtering done), and assuming the message is always from root, the rank should be 0:

axom/src/axom/slic/streams/SynchronizedStream.cpp

Lines 145 to 152 in 6f5eaa3

	m_cache->messages.push_back(
	this->getFormatedMessage(message::getLevelAsString(msgLevel),
	message,
	tagName,
	std::to_string(rank),
	"1",
	fileName,
	line));

---

---

Lumberjack streams now have the trailing dots "..." for the rank list, even if the rank count is 1. Is it possible at all to disable the dots so that I just get the first number in the rank list? This would also help with consistency in formatting.

I'm not too sure where this is coming from. @white238, could it be this line here in Lumberjack (line 58)?

axom/src/axom/lumberjack/Message.cpp

Lines 42 to 61 in 6f5eaa3

	std::string Message::stringOfRanks(std::string delimiter) const
	{
	std::string returnString = "";
	int ranksSize = m_ranks.size();
	for(int i = 0; i < ranksSize; ++i)
	{
	returnString += std::to_string(m_ranks[i]);
	if(i < (ranksSize - 1))
	{
	returnString += delimiter;
	}
	}
	// Indicate we might have more ranks we aren't individually tracking
	if(m_ranksLimitReached)
	{
	returnString += "...";
	}
	return returnString;
	}

[gberg617]

I had tried the SynchronizedStream previously, the formatting for this worked well. From a conceptual standpoint, I wasn't sure if using this makes sense if it is only used to output messages from a single rank (the root rank). I noticed that the flush method for SynchronizedStream has some MPI logic to check for messages that need to be sent or received. Would these be called when I'm just expecting a single rank to output messages? If so, would there be any performance implications for calling these MPI routines?

[bmhan12]

I had tried the SynchronizedStream previously, the formatting for this worked well. From a conceptual standpoint, I wasn't sure if using this makes sense if it is only used to output messages from a single rank (the root rank). I noticed that the flush method for SynchronizedStream has some MPI logic to check for messages that need to be sent or received. Would these be called when I'm just expecting a single rank to output messages? If so, would there be any performance implications for calling these MPI routines?

I see, that's a good point. You likely would see some costs for those MPI routines.

Alternatively, if you know for sure the GenericOutputStream will be used only for the root node, you could use a constructor with the format parameter to hardcode the rank/rank count information.

For example, in this Lumberjack format string example:

axom/src/axom/slic/examples/basic/lumberjack_logging.cpp

Lines 41 to 58 in 6f5eaa3

	// Initialize SLIC
	constexpr const char* format = R"(
	<MESSAGE>
	\t<TIMESTAMP>
	\tLEVEL=<LEVEL>
	\tRANKS=<RANK>
	\tRANK_COUNT=<RANK_COUNT>
	\tFILE=<FILE>
	\tLINE=<LINE>
	)";
	slic::initialize();
	// Set SLIC logging level and Lumberjack Logging stream
	slic::setLoggingMsgLevel(slic::message::Debug);
	slic::disableAbortOnError();
	slic::LumberjackStream* ljStream =
	new slic::LumberjackStream(&std::cout, MPI_COMM_WORLD, RANKSLIMIT, format);
	slic::addStreamToAllMsgLevels(ljStream);

You can instead hardcode the rank and rank count:

  // Initialize SLIC
  constexpr const char* format = R"(
<MESSAGE>
\t<TIMESTAMP>
\tLEVEL=<LEVEL>
\tRANKS=0
\tRANK_COUNT=1
\tFILE=<FILE>
\tLINE=<LINE>
)";

EDIT: Essentially, you can use these constructors with the format parameter to customize your GenericOutputStream to include your rank information:

axom/src/axom/slic/streams/GenericOutputStream.hpp

Lines 58 to 82 in 6f5eaa3

	/*!
	* \brief Constructs a GenericOutputStream instance with the given stream and
	* message formatting.
	* \param [in] os pointer to a user-supplied ostream instance.
	* \param [in] format the format string.
	* \pre os != NULL
	* \see LogStream::setFormatString for the format string.
	*/
	GenericOutputStream(std::ostream* os, const std::string& format);
	/*!
	* \brief Constructs a GenericOutputStream instance specified by the given
	* string and message formatting.
	* The string input determines the stream as follows:
	* - "cout" makes std::cout the output stream
	* - "cerr" makes std::cerr the output stream
	* - Any other input will construct a std::ofstream associated with input
	* \param [in] stream the string to control type of stream created
	* \param [in] format the format string.
	* \see LogStream::setFormatString for the format string.
	*
	* \note This constructor avoids creating an empty file if this
	* GenericOutputStream never flushes a message.
	*/
	GenericOutputStream(const std::string& stream, const std::string& format);

[gberg617]

This is an interesting point. The only formatting keyword I generally pass to Slic is <MESSAGE> because I find it easier to format the full message in the application code itself. This also makes it easier to embed certain variables within the message that don't have Slic keywords associated with them such as simulation time and simulation cycle. As a bit of a side note, this comes at a cost of having to send longer messages when using the Lumberjack stream (since it includes all the formatting), but I don't really know how else to get around this issue.

Perhaps to enforce rank=0, I can check which stream is used when I format my messages and override the <RANK> keyword for the string 0. This could be a viable alternative. UPDATE: This does work!

[bmhan12]

@gberg617

@white238 had some suggestions on point 2) with the trailing dots (please correct me if I am misremembering):

• The ranksLimit field may be set lower than expected, causing the trailing dots to appear. May make sense to bump it higher:

  axom/src/axom/slic/streams/LumberjackStream.hpp

  Lines 50 to 59 in e5dcbaa

  	/*!
  	* \brief Constructs a LumberjackStream instance with the given stream,
  	* MPI communicator, and rank limit.
  	* \param [in] stream pointer to a user-supplied ostream instance.
  	* \param [in] comm MPI communicator
  	* \param [in] ranksLimit limit on how many ranks are individually tracked per
  	* message
  	* \pre stream != NULL
  	*/
  	LumberjackStream(std::ostream* stream, MPI_Comm comm, int ranksLimit);

• Potentially using/implementing a version of SLIC_X_ROOT/SLIC_X_IF macros may be useful in isolating the rank you want printed if the issue is bumping up against the ranksLimit:
  • List of Axom macros: https://axom.readthedocs.io/en/develop/axom/slic/docs/sphinx/sections/appendix.html
  • Axom macros definitions: https://github.com/LLNL/axom/blob/develop/src/axom/slic/interface/slic_macros.hpp
  • EDIT: Additionally, here is unit tests showing the trailing dots behavior:

    axom/src/axom/lumberjack/tests/lumberjack_Message.hpp

    Lines 389 to 443 in e5dcbaa

    	TEST(lumberjack_Message, stringOfRanks02)
    	{
    	//Test case: more than one rank
    	const int ranksLimit = 5;
    	std::vector<int> ranks;
    	for(int i = 0; i < (int)ranksLimit; ++i)
    	{
    	ranks.push_back(i + 1);
    	}
    	axom::lumberjack::Message m;
    	m.addRanks(ranks, ranksLimit, ranksLimit);
    	EXPECT_EQ(m.text(), "");
    	EXPECT_EQ(m.fileName(), "");
    	EXPECT_EQ(m.lineNumber(), 0);
    	EXPECT_EQ(m.ranks().size(), (std::vector<int>::size_type)ranksLimit);
    	EXPECT_EQ(m.count(), ranksLimit);
    	for(int i = 0; i < (int)ranksLimit; ++i)
    	{
    	EXPECT_EQ(m.ranks()[i], i + 1);
    	}
    	EXPECT_EQ(m.stringOfRanks(), "1,2,3,4,5...");
    	}
    	TEST(lumberjack_Message, stringOfRanks03)
    	{
    	//Test case: full Message
    	const int ranksLimit = 5;
    	std::vector<int> ranks;
    	for(int i = 0; i < (int)ranksLimit; ++i)
    	{
    	ranks.push_back(i * 2);
    	}
    	axom::lumberjack::Message m("Unimportant message",
    	ranks,
    	ranksLimit,
    	ranksLimit,
    	"test/foo.cpp",
    	987654321,
    	0,
    	"");
    	EXPECT_EQ(m.text(), "Unimportant message");
    	EXPECT_EQ(m.fileName(), "test/foo.cpp");
    	EXPECT_EQ(m.lineNumber(), 987654321);
    	EXPECT_EQ(m.ranks().size(), (std::vector<int>::size_type)ranksLimit);
    	EXPECT_EQ(m.count(), ranksLimit);
    	for(int i = 0; i < (int)ranksLimit; ++i)
    	{
    	EXPECT_EQ(m.ranks()[i], i * 2);
    	}
    	EXPECT_EQ(m.stringOfRanks(), "0,2,4,6,8...");
    	}

[bmhan12]

@gberg617 Closing this issue for now. Feel free to re-open this issue/open a new issue if the problem persists!