images list API, avoid multiple prints of duplicate images.

[fengwei0328]

What is the problem you're trying to solve

Due to the CRI interface, whether pulling images or loading images, the CRI interface writes data to the meta.db again.
The storage methods between CRI and containerd are inconsistent:
1 The imageStore on the CRI side performs deduplication,
2 The imageStore on the ctr side reads directly from the database.
This leads to the presence of duplicate entries when viewing images in the k8s.io namespace, which affects the user experience when there are a large number of images.

cri:

$ crictl images       
IMAGE                      TAG                 IMAGE ID            SIZE
docker.io/library/centos   7                   eeb6ee3f44bd0       76.1MB

nerdctl:

REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM       SIZE         BLOB SIZE
centos        7         be65f488b776    59 minutes ago    linux/amd64    211.5 MiB    72.6 MiB
centos        <none>    be65f488b776    59 minutes ago    linux/amd64    211.5 MiB    72.6 MiB
<none>        <none>    be65f488b776    59 minutes ago    linux/amd64    211.5 MiB    72.6 MiB

ctr:

REF                                                                                              TYPE                                                      DIGEST                                                                  SIZE     PLATFORMS                                                       LABELS                          
docker.io/library/centos:7                                                                       application/vnd.docker.distribution.manifest.list.v2+json sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4 72.6 MiB linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8,linux/ppc64le io.cri-containerd.image=managed 
docker.io/library/centos@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4 application/vnd.docker.distribution.manifest.list.v2+json sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4 72.6 MiB linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8,linux/ppc64le io.cri-containerd.image=managed 
sha256:eeb6ee3f44bd0b5103bb561b4c16bcb82328cfe5809ab675bb17ab3a16c517c9                          application/vnd.docker.distribution.manifest.list.v2+json sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4 72.6 MiB linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8,linux/ppc64le io.cri-containerd.image=managed

The above indicates that when using ctr -n k8s.io i import image.tar to loading an image, CRI will use the config ID to load it again; if it's during the pull, it will load three duplicate entries.
https://github.com/containerd/containerd/blob/9f8dfdc280682a41b36e6e7ee2f4646151fb6861/internal/cri/server/images/image_pull.go#L224-L230

Pulling an image will generate three entries.
https://github.com/containerd/containerd/blob/566f9f48db6f6bc7dbd76d70aa425afbee7b0d0a/internal/cri/server/images/image_pull.go#L238

Therefore, the following phenomenon will occur.

$ nerdctl -n k8s.io images
REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM       SIZE         BLOB SIZE
centos        7         be65f488b776    44 minutes ago    linux/amd64    211.5 MiB    72.6 MiB
centos        <none>    be65f488b776    44 minutes ago    linux/amd64    211.5 MiB    72.6 MiB
<none>        <none>    be65f488b776    44 minutes ago    linux/amd64    211.5 MiB    72.6 MiB

When using nerdctl or ctr to delete an image, I have to delete it three times and cannot delete it by specifying the image ID!!

nerdctl -n k8s.io rmi be65f488b776          
FATA[0000] 1 errors:
multiple IDs found with provided prefix: be65f488b776

Describe the solution you'd like

Currently, there is a approaches to solve this issue:
Deduplicate images with the same digest during the list query.
The nerdctl images command has an --all parameter that is configured to be true by default but is not currently utilized. I would like to use it as a parameter to display duplicate images. Adding the --all parameter will show all tags of an image, while omitting --all will display only a single entry. By default, --all is set to false.

nerdctl/cmd/nerdctl/image/image_list.go

Line 68 in 1dd11e1

imagesCommand.Flags().BoolP("all", "a", true, "(unimplemented yet, always true)")

imagesCommand.Flags().BoolP("all", "a", false, "Show all images repo, include imageID, repoTAg, repoDigest")

nerdctl/pkg/cmd/image/list.go

Lines 129 to 186 in 1dd11e1

	func printImages(ctx context.Context, client *containerd.Client, imageList []images.Image, options *types.ImageListOptions) error {
	w := options.Stdout
	digestsFlag := options.Digests
	if options.Format == "wide" {
	digestsFlag = true
	}
	var tmpl *template.Template
	switch options.Format {
	case "", "table", "wide":
	w = tabwriter.NewWriter(w, 4, 8, 4, ' ', 0)
	if !options.Quiet {
	printHeader := ""
	if options.Names {
	printHeader += "NAME\t"
	} else {
	printHeader += "REPOSITORY\tTAG\t"
	}
	if digestsFlag {
	printHeader += "DIGEST\t"
	}
	printHeader += "IMAGE ID\tCREATED\tPLATFORM\tSIZE\tBLOB SIZE"
	fmt.Fprintln(w, printHeader)
	}
	case "raw":
	return errors.New("unsupported format: \"raw\"")
	default:
	if options.Quiet {
	return errors.New("format and quiet must not be specified together")
	}
	var err error
	tmpl, err = formatter.ParseTemplate(options.Format)
	if err != nil {
	return err
	}
	}
	printer := &imagePrinter{
	w: w,
	quiet: options.Quiet,
	noTrunc: options.NoTrunc,
	digestsFlag: digestsFlag,
	namesFlag: options.Names,
	tmpl: tmpl,
	client: client,
	provider: containerdutil.NewProvider(client),
	snapshotter: containerdutil.SnapshotService(client, options.GOptions.Snapshotter),
	}
	for _, img := range imageList {
	if err := printer.printImage(ctx, img); err != nil {
	log.G(ctx).Warn(err)
	}
	}
	if f, ok := w.(formatter.Flusher); ok {
	return f.Flush()
	}
	return nil
	}

func printImages(ctx context.Context, client *containerd.Client, imageList []images.Image, options types.ImageListOptions) error {
	w := options.Stdout
	var ImageList []images.Image
	if options.All {
		ImageList = imageList
	} else {
		var image_digest = make(map[string]bool)
		for _, ima := range imageList {
			if image_digest[string(ima.Target.Digest)] || strings.HasPrefix(ima.Name, "sha256:") {
				continue
			}
			ImageList = append(ImageList, ima)
			image_digest[string(ima.Target.Digest)] = true
		}
	}
        //......
       for _, img := range ImageList {
		if err := printer.printImage(ctx, img); err != nil {
			log.G(ctx).Warn(err)
		}
	}
        //.............
}

all tags will only be displayed when the "all" parameter is included. If it is not added, only the image name will be shown.

Additional context

Comparison of effects before and after modification:

# nerdctl -n k8s.io images -a
REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM       SIZE         BLOB SIZE
centos        7         be65f488b776    38 minutes ago    linux/amd64    211.5 MiB    72.6 MiB
centos        <none>    be65f488b776    39 minutes ago    linux/amd64    211.5 MiB    72.6 MiB
<none>        <none>    be65f488b776    39 minutes ago    linux/amd64    211.5 MiB    72.6 MiB

# nerdctl -n k8s.io images 
REPOSITORY    TAG    IMAGE ID        CREATED              PLATFORM       SIZE         BLOB SIZE
centos        7      be65f488b776    About an hour ago    linux/amd64    211.5 MiB    72.6 MiB

[apostasie]

@fengwei0328 what would happen when you have multiple legit images with different tags and the same digest?

nerdctl images
REPOSITORY    TAG       IMAGE ID        CREATED              PLATFORM       SIZE       BLOB SIZE
foofoo        latest    c4561867435f    1 second ago         linux/arm64    402.4MB    118.2MB
mariadb       10.5      c4561867435f    About an hour ago    linux/arm64    402.4MB    118.2MB

[fengwei0328]

@fengwei0328 what would happen when you have multiple legit images with different tags and the same digest?

nerdctl images
REPOSITORY    TAG       IMAGE ID        CREATED              PLATFORM       SIZE       BLOB SIZE
foofoo        latest    c4561867435f    1 second ago         linux/arm64    402.4MB    118.2MB
mariadb       10.5      c4561867435f    About an hour ago    linux/arm64    402.4MB    118.2MB

@apostasie
This is somewhat different from re-tagging. It is an entry that is automatically written when loading or pulling an image under the k8s.io namespace. When I tried to delete the image using nerdctl, I encountered an issue because there were two entries with both repo/tag fields set to <none>, making it impossible to delete. I needed to use ctr to check the image reference before deletion.

$ nerdctl -n k8s.io images
REPOSITORY          TAG       IMAGE ID        CREATED               PLATFORM       SIZE         BLOB SIZE
centos              7         be65f488b776    About a minute ago    linux/amd64    211.5 MiB    72.6 MiB
centos              <none>    be65f488b776    About a minute ago    linux/amd64    211.5 MiB    72.6 MiB
<none>              <none>    be65f488b776    About a minute ago    linux/amd64    211.5 MiB    72.6 MiB

$ nerdctl -n k8s.io rmi centos:7          
Untagged: docker.io/library/centos:7@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4
Deleted: sha256:174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02

 nerdctl -n k8s.io images          
REPOSITORY          TAG       IMAGE ID        CREATED          PLATFORM       SIZE         BLOB SIZE
centos              <none>    be65f488b776    2 minutes ago    linux/amd64    211.5 MiB    72.6 MiB
<none>              <none>    be65f488b776    2 minutes ago    linux/amd64    211.5 MiB    72.6 MiB

I need to get the Ref on the ctr side to delete it, and complete the last two cleanups, so that a centos:7 image can be completely cleaned up

 ctr -n k8s.io i ls
REF                                                                                              TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                       LABELS                          
docker.io/library/centos@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4 application/vnd.docker.distribution.manifest.list.v2+json sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4 72.6 MiB  

sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4 72.6 MiB  linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8,linux/ppc64le io.cri-containerd.image=managed 

I need to delete Ref from ctr

$ nerdctl -n k8s.io rmi docker.io/library/centos@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4        
Untagged: docker.io/library/centos@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4
Deleted: sha256:174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02

Finally, delete the last entry using the image ID.

$ nerdctl -n k8s.io rmi be65f488b776                                                                                                
Untagged: sha256:eeb6ee3f44bd0b5103bb561b4c16bcb82328cfe5809ab675bb17ab3a16c517c9@sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4
Deleted: sha256:174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02

[apostasie]

Thanks @fengwei0328

I was not clear. My comment should have read: "with your proposed change, what would happen to...".

Anyhow, this here is certainly a problem.
I can't work on this right now, but if nobody volunteers in the meantime, I'll have a look later on.

[fengwei0328]

I was not clear. My comment should have read: "with your proposed change, what would happen to...".

I want to modify it so that when I run nerdctl -n k8s.io images to view images under k8s.io, there is only one image displayed in the results. Also, when deleting the image, I should only need to delete it once to achieve a complete cleanup.
Currently, the issue I encountered while trying to make this modification is that if an image generated using nerdctl tag is present, it will be deleted together.