Grammar-Assisted Generation with LLama-cpp-python: A Step-by-Step Guide

Introducing Grammar-Assisted Generation for LLMs

The latest update to the llama-cpp-python library for LLMs has introduced a groundbreaking feature that allows for the insertion of custom code into the model’s output generation process. This technique, known as grammar-assisted generation, ensures that the overall output exactly matches the grammar specified. The author provides a detailed guide on how to use this feature with the JSON grammar, which guarantees a valid JSON array output. However, the author also highlights a limitation of this technique: the possibility of incomplete JSON output due to token truncation. To overcome this limitation, the author suggests using ijson, a streaming JSON parser.

The author also mentions other tools and resources, such as Grammar Builder by Intrinsic Labs and the llama.cpp script examples/json-schema-to-grammar.py, that can help generate grammars from TypeScript declarations or JSON schemas. This feature has the potential to enable the creation of LLM-based functions that can run on personal devices, such as OpenAI Functions on top of models like Llama 2.

A Simple “Be My Eyes” Web App with a Llama.cpp/Llava Backend

Inspired by Fuzzy-Search/realtime-bakllava, the author has created a simple “Be My Eyes” web app with a Llama.cpp/Llava backend. The app uses the SkunkworksAI BakLLaVA-1 model via a Llama.cpp backend and narrates its descriptions using the Web Speech API. The author provides instructions on how to set up the Llama.cpp server and build the app.

Conclusion

Grammar-assisted generation is a powerful technique that allows for more precise and controlled output from LLMs, making them more useful for various applications. The author’s guide on how to use this feature with the JSON grammar, along with the suggested tools and resources, provides a solid foundation for exploring this exciting new development in LLM technology. The author’s “Be My Eyes” web app is a testament to the potential of LLMs to bridge gaps and create opportunities, making the web a more accessible place for the visually impaired.