CsvReader and memory release

[agent1red]

I noticed that when you run a method over again when using CsvDataReader the Process memory dramatically increases. Am I missing something here? I called csv.Close() after the ReadAsync() operation but nothing happens.

[MarkPflug]

I cannot reproduce this behavior based on your minimal description. I can't guess what's going on without seeing some code.

[agent1red]

Thanks for responding so quickly. So I am creating a new model when reading the csv file with many columns (50 plus) with many rows of data (about 7,000) all is good except the memory usage particularly with csv.GetString() or other types. I am just setting this up in a blazor project with latest .net etc.. I have this set up to call these Get methods upon OnInitialize (page load). When I refresh, the memory continues to stack ( about 230 mb) each time. I was able to get a work around with garbage collector and I can dump those objects immediately after. I also tried the old way of using statement. Just noticed that maybe we have to handle memory usage ourselves. I have used excel interop and realized you have to run garbage collector as well after opening a workbook and closing it. BTW what is the main use of CsvDataReader.Close()? I am wondering if I am using this correctly.

Original way - following examples.

// inside method
using var csv = CsvDataReader.Create(FilePath); // Getting a csv from file path here 

while(await csv.ReadAsync()) 
{
	someModel.Add( new someModel()
	{
		id = csv.GetInt32(0);
		name = csv.GetString(1);
		date = csv.GetDateTime(2);
		///50 more columns
	}
    csv.Close();
}

What I did to alleviate memory issue with GC. Can be in main OnInitialized method instead.

 // inside method

 using (CsvDataReader csv = CsvDataReader.Create(FilePath))
            {
                while (await csv.ReadAsync())
                {
                    someModel.Add(new someModel()
                    {
                        id = csv.GetInt32(0);
			name = csv.GetString(1);
			date = csv.GetDateTime(2);
                        ///50 more columns
                    });
                }
                csv.Close();
            }
GC.Collect();
GC.WaitForPendingFinalizers();

return someModel;

[MarkPflug]

So, this is a Blazor Server app? I believe what you are seeing is due to "Server GC" mode. This is the default mode for server applications, which includes asp.net apps. In that mode the GC doesn't run until there is memory pressure. I think the behavior you are seeing is normal.

However, I don't know what someModel is, or what you are doing with it. If your application holds on to the someModel collection in some way, then it won't ever get GCed. That would be a flaw in your application code though, not in the CSV library.

I would advise against calling GC methods (collect, etc) manually. I think the guidance is that those methods are intended for memory analysis, performance and benchmarking purposes, and shouldn't be called by normal application code.

[MarkPflug]

Oh, and calling Close does the same thing as Dispose (using). You don't need to call Close as the using statement already handles that.

[agent1red]

Awesome. I figured that was the case for Close. Yes, this is a Blazor server app. The someModel is meant for the csv file column headers (as properties of that model). Need to pull in data from a csv file, do my changes to some of its data and save it to a database. I haven't gotten to that part yet where GC might happen. Just cringed when I saw the page refresh and the memory going up drastically.