eleticle
Tag · Deployment

Model serving and efficiency

Running a model fast and cheaply enough: inference speed, memory, quantisation and serving. 1 entry in the Core Log.

  1. No. 07Nov 2026

    What hardware does a mineral-processing site need to self-host LLMs and expert models?

    An A$43,568 machine with one 96 GB card, plus a 16 GB tools card, runs GLM-5.3-Flash at an estimated 56 tokens per second for one 128K-token conversation. For the large expert models, RAM and the software stack matter: on one computer, the right stack, with its default drafting, wrote three to five times faster.