Pagination and permalinks with more complex/grouped data

[furyd]

Hi everyone,

My requirement is to generate a list of records with the permalink pattern of /items/{{item.type}}-near{{item.area}}/page-{{pagination.pageNumber}} with a page size of 20 items.

My recordset looks like this:

id | type | area | name |...

I'm currently grouping the data via using a Map with the type and area as the key, then converting that into an array with the shape of:

[
    {
        role: 'role',
        area: 'area',
        data: [{}.....]
    }
]

And the test liquid file is like so:

---
layout: layouts/general.liquid
pagination:
    data: test
    size: 20
    alias: provider
permalink: "testing/{{provider.role}}/{{provider.area}}/{{pagination.pageNumber}}/"
---

<h2>{{provider.id}} | {{provider.name}}</h2>

Which isn't doing what I would like it to do.

Any suggestions and advice appreciated!

Thanks in advance

[furyd]

Okay, I managed to achieve this with collections after finding a link suggesting that, which is a bit hacky, but that's fine, it's still early days on what's already the best static generator in my not very humble opinion!

However there's a definite issue with processing large volumes of data, underscore consumed all the memory and understandably triggered an error, killing the page generation when there were north of 10k records to create/

[pdehaan]

Yeah, it sounds like you're describing #461 (double layered pagination).

I think the best solution I've seen so far is to pre-create custom collections, if possible. Otherwise, pre-generating global data files might be an option, but it depends on how dynamic your data is. It might be tricky if you're trying to build some Yelp-like site where you want to paginate over all restaurants near the St Louis area and have a big data set.

[furyd]

Revisited this, my solution was to have a powershell script get the first layer, then replace tokens inside a liquid template and js file, then run eleventy.

It's quick, flexible and effective and (so far) removes the large volume issues I hit.

[pdehaan]

@furyd What does the powershell script look like?

[furyd]

@furyd What does the powershell script look like?

It's just a simple script that loops through a recordset, and does a string replace on some tokens before creating the liquid and data files:

Install-Module -Name SqlServer -Force

$roles = Invoke-Sqlcmd -ServerInstance "<<server>>" -Database "<<db>>" -Username "<<user>>" -Password "<<pwd>>" -Query "EXEC [dbo].[ListRoles]";

$sourceDirectory = "$($PSScriptRoot)\templates";
$sourceDataDirectory = "$($sourceDirectory)\_data";
$targetDirectory = $PSScriptRoot;
$targetDataDirectory = "$($targetDirectory)\_data";

if(!(Test-Path $targetDataDirectory)){
    New-Item -Path $targetDataDirectory -ItemType Directory
}

foreach ($role in $roles) {

    $targetTemplate = "$($targetDirectory)\provider.liquid";
    $targetData = "$($targetDataDirectory)\providers.js"

    if(Test-Path $targetTemplate){
        Remove-Item $targetTemplate;
    }

    if(Test-Path $targetData){
        Remove-Item $targetData;
    }

    $content = Get-Content -Path "$($sourceDirectory)\provider.liquid";
    Set-Content -Path $targetTemplate -Value $content.replace("#{role}#", $role.Name);

    $content = Get-Content -Path "$($sourceDataDirectory)\providers.js";
    Set-Content -Path $targetData -Value $content.replace("#{role-id}#", $role.Id);

    npm run build
}

Where npm run build triggers eleventy.

I'm using an .eleventyignore so the templates folder isn't processed.

It can be expanded on quite a bit, e.g. if you need parallelism or isolated folders, and use dir:output to combine.