---
sd_hide_title: true
---

# 🔎 OptiML Overview

::::{grid}
:reverse:
:gutter: 3 4 4 4
:margin: 1 2 1 2

:::{grid-item}
:columns: 12 4 4 4

```{image} _static/logo-square.png
:width: 200px
:class: sd-m-auto
:name: landing-page-logo
```

:::

:::{grid-item}
:columns: 12 8 8 8
:child-align: justify
:class: sd-fs-5

```{rubric} OptiML: Drop-in inference engine for AI agents.
```

OptiML, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. 
The key underlying the design of OptiML is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation.

````{div} sd-d-flex-row
```{button-ref} intro
:ref-type: doc
:color: primary
:class: sd-rounded-pill sd-mr-3

Get Started
```

```{button-ref} demo
:ref-type: doc
:color: secondary
:class: sd-rounded-pill

OptiML Demo
```
````

:::

::::

---

::::{grid} 1 2 2 3
:gutter: 1 1 1 2

:::{grid-item-card} {octicon}`gear;1.8em;sd-mr-1` Neuron-Aware Offloading

Hot neuron run on GPU, cold neurons on CPU. Breaks the shackle of VRAM size.

:::

:::{grid-item-card} {octicon}`plug;1.8em;sd-mr-1` Adaptive Activation Predictors

Learn your model's pattern dynamically, adaptively. Ensures maximum compatibility.

:::

:::{grid-item-card} {octicon}`tools;1.8em;sd-mr-1` Sparse Neuron Operators

Only calculate what's needed. No wasted work on useless data.

:::

:::{grid-item-card} {octicon}`dependabot;1.8em;sd-mr-1` Drop-in Integration

Can be applied to all transformer-based models. Accelerates your model effortlessly.

:::

:::{grid-item-card} {octicon}`infinity;1.8em;sd-mr-1` Multi-language Bindings

Supports major programming languages. Front-end libraries available for quick access.

:::

:::{grid-item-card} {octicon}`globe;1.8em;sd-mr-1` Corporate Collaboration

Teamed up with leading tech companies. Continuously adopts bleeding-edge technology.

:::

::::

---
<!-- 
```{rubric} TODO
```

[MyST-Markdown VS Code extension](https://marketplace.visualstudio.com/items?itemName=ExecutableBookProject.myst-highlight)
: For MyST extended syntax highlighting and authoring tools.

[Convert existing ReStructuredText files to Markdown][rst-to-myst]
: Use the [rst-to-myst] CLI or [the MySTyc interactive web interface](https://astrojuanlu.github.io/mystyc/).

[markdown-it-py]
: A CommonMark-compliant and extensible Markdown parser, used by MyST-Parser to parse source text to tokens.
-->

```{rubric} Acknowledgements
```
The OptiML project was initiated at QRG lab, Northwestern University. In the project's early stage, we received contributions from top minds at leading institutions around the world. Special thanks to them who made this project possible!

<table>
  <tr>
    <th colspan="4" style="text-align:center">
    <img src="_static/assets/nvidia.png" width="64" height="36" style="display:block; margin:auto; margin-top:5px; margin-bottom:-1px">
    NVIDIA
    </th>
  </tr>
  <tr>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/gauravjuvekar.png" width="128" height="128"><br><a href=https://github.com/gauravjuvekar>Gaurav Juvekar</a>
    </td>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/rg20.png" width="128" height="128"><br><a href=https://github.com/rg20>Rajesh Gandham</a>
    </td>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/akifcorduk.png" width="128" height="128"><br><a href=https://github.com/akifcorduk>Akif Corduk</a>
    </td>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/booxter.png" width="128" height="128"><br><a href=https://github.com/booxter>Ihar Hrachyshka</a>
    </td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center">
    <img src="_static/assets/meta.png" width="64" height="36" style="display:block; margin:auto; margin-top:5px; margin-bottom:-1px;">
    Meta
    </th>
  </tr>
  <tr>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/ashwinb.png" width="128" height="128"><br><a href=https://github.com/ashwinb>Ashwin Bharambe</a>
    </td>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/dltn.png" width="128" height="128"><br><a href=https://github.com/dltn>Dalton Flanagan</a>
    </td>
    <td style="text-align:center;"></td>
    <td style="text-align:center;"></td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center">
    <img src="_static/assets/redhat.png" width="64" height="36" style="display:block; margin:auto; margin-top:5px; margin-bottom:-1px;">
    Red Hat
    </th>
  </tr>
  <tr>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/leseb.png" width="128" height="128"><br><a href=https://github.com/leseb>Sebastien Han</a>
    </td>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/cdoern.png" width="128" height="128"><br><a href=https://github.com/cdoern>Charlie Doern</a>
    </td>
    <td style="text-align:center;"></td>
    <td style="text-align:center;"></td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center">
    <img src="_static/assets/purdue.png" width="64" height="36" style="display:block; margin:auto; margin-top:5px; margin-bottom:-1px;">
    Purdue University
    </th>
  </tr>
  <tr>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/bernalde.png" width="128" height="128"><br><a href=https://github.com/bernalde>David Bernal</a>
    </td>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/jihunhwang.png" width="128" height="128"><br><a href=https://github.com/jihunhwang>Jihun Hwang</a>
    </td>
    <td style="text-align:center;"></td>
    <td style="text-align:center;"></td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center">
    <img src="_static/assets/ucsd.png" width="64" height="36" style="display:block; margin:auto; margin-top:5px; margin-bottom:-1px;">
    UCSD
    </th>
  </tr>
  <tr>
    <td style="text-align:center;">
      <img src="_static/assets/profiles/dothyt.png" width="128" height="128"><br><a href=https://github.com/dothyt>Yutong Huang</a>
    </td>
    <td style="text-align:center;"></td>
    <td style="text-align:center;"></td>
    <td style="text-align:center;"></td>
  </tr>
  <tr>
    <th colspan="4" style="text-align:center">
    <a href=#><img src="_static/assets/community.png" width="64" height="36" style="display:block; margin:auto; margin-bottom:8px;"></a>
    Community
    </th>
  </tr>
  <tr>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/alugupta.png" width="128" height="128"></a>
      <br>Udit Gupta
    </td>
        <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/SirenaYu.png" width="128" height="128"></a>
      <br>Sirena Yu
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/JandaTom.png" width="128" height="128"></a>
      <br>Tomas Janda
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/MartinoMensio.png" width="128" height="128"></a>
      <br>Martino Mensio
    </td>
  </tr>
  <tr>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/egerber.png" width="128" height="128"></a>
      <br>Emanuel Gerber
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/nour-taqatqa.png" width="128" height="128"></a>
      <br>Nour Taqatqa
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/SungMinCho.png" width="128" height="128"></a>
      <br>Sung Min Cho
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/Hibb-bb.png" width="128" height="128"></a>
      <br>Dennis Wu
    </td>
  </tr>
  <tr>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/henryburon.png" width="128" height="128"></a>
      <br>Henry Buron
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/gkoundry.png" width="128" height="128"></a>
      <br>Glen Koundry
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/jimbo797.png" width="128" height="128"></a>
      <br>Jimmy Kuhlman
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/devanshuDesai.png" width="128" height="128"></a>
      <br>Devanshu Desai
    </td>
  </tr>
  <tr>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/brianjsl.png" width="128" height="128"></a>
      <br>Brian Lee
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/MarkoSterbentz.png" width="128" height="128"></a>
      <br>Marko Sterbentz
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/sam-leeman.png" width="128" height="128"></a>
      <br>Sam Leeman
    </td>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/barrieca.png" width="128" height="128"></a>
      <br>Cameron Barrie
    </td>
  </tr>
  <tr>
    <td style="text-align:center;">
      <a href=#><img src="_static/assets/profiles/k8sify.png" width="128" height="128"></a>
      <br>k8sify
    </td>
    <td style="text-align:center;"></td>
    <td style="text-align:center;"></td>
    <td style="text-align:center;"></td>
  </tr>

  
</table>

```{toctree}
:hidden:
:caption: ✏️ GET STARTED

intro.md
demo.md
optiml_apis.md
optiml_works.md
```

```{toctree}
:hidden:
:caption: 📚 GUIDANCE

architecture.md
build.md
faq/index.md
```

```{toctree}
:hidden:
:caption: 📖 OPTIML COMMUNITY

syntax/roadmap.md
develop/index.md
```

[commonmark]: https://commonmark.org/
[github-ci]: https://github.com/executablebooks/MyST-Parser/workflows/continuous-integration/badge.svg?branch=master
[github-link]: https://github.com/executablebooks/MyST-Parser
[codecov-badge]: https://codecov.io/gh/executablebooks/MyST-Parser/branch/master/graph/badge.svg
[codecov-link]: https://codecov.io/gh/executablebooks/MyST-Parser
[rtd-badge]: https://readthedocs.org/projects/myst-parser/badge/?version=latest
[rtd-link]: https://myst-parser.readthedocs.io/en/latest/?badge=latest
[black-badge]: https://img.shields.io/badge/code%20style-black-000000.svg
[pypi-badge]: https://img.shields.io/pypi/v/myst-parser.svg
[pypi-link]: https://pypi.org/project/myst-parser
[conda-badge]: https://anaconda.org/conda-forge/myst-parser/badges/version.svg
[conda-link]: https://anaconda.org/conda-forge/myst-parser
[black-link]: https://github.com/ambv/black
[github-badge]: https://img.shields.io/github/stars/executablebooks/myst-parser?label=github
[markdown-it-py]: https://markdown-it-py.readthedocs.io/
[markdown-it]: https://markdown-it.github.io/
[rst-to-myst]: https://rst-to-myst.readthedocs.io