Memory Requirements and iOS Support?

[Lukas1h]

Dose this run on mobile devices? If so, what are the memory requierments?

[liuliu]

It should, haven't tried.

Consumes about 4GiB.

[Lukas1h]

Thanks. How would one go about converting the model to fp16?

[liuliu]

You can load f32 weights into a f16 model and then write the f16 model weights out. Current txt2img/main.swift example loads f32 weights into f16 model. You can then just add additional:

graph.openStore("xxx.ckpt") {
  $0.write("unet", model: unet)
  $0.write("decoder", model: decoder)
}

and the new file will be f16 weights.

[Lukas1h]

Would this decrease memory usage under 3gb? Thanks again for the help!

[liuliu]

No. Model already running at f16 with f16 weights. It just affects files stored on disk. FP16 already runs at around 3.3GiB including all the models (text model, unet and decoder). The extra 0.7 is from MPS implicit tensors which can be further reduced by just running one op at a time (slower).

[Lukas1h]

Have the memory requirements changed since being published on the App Store?
This issue seems to lead me to believe so.