Six AI Engineers Share Insights from a Year of Building with Large Language Models

Practical Tips and Experiences in Developing Real-World Applications with LLMs, Garnering High Praise from the Developer Community

Summary

Six AI engineers share a year’s worth of hands-on experience with large language models, offering valuable insights and practical tips.

(AIM)—Six frontline AI engineers and entrepreneurs have shared their hard-earned insights from a year of building applications with large language models (LLMs). This comprehensive and practical long-read has quickly become a hot topic in the developer community. Praised for its actionable advice, it’s a must-read for anyone working with LLMs.

These six authors come from diverse backgrounds, including big tech engineers, independent developers, and consultants. Despite their different roles, they all spent the past year building real applications on LLMs, not just flashy demos. They emphasize that now is the time for even non-machine learning engineers or scientists to integrate AI into their products. Here are some of the highlights and practical tips that have sparked lively discussions among developers:

When to Use Long Contexts, RAG, or Fine-Tuning

• Diverse Outputs: Adjusting example sequences in prompts can significantly affect results, not just by increasing temperature.
• Intern Test: If an intern can complete the task based on your prompt, it’s probably well-structured.
• Model Preferences: Each LLM has its preferences, such as Claude favoring XML and GPT series favoring Markdown and JSON.
• Cost-Effectiveness: If a prompt achieves 90% of the task, fine-tuning may not be worth the investment.

Prompts, RAG, and Fine-Tuning

Prompts, RAG (retrieval-augmented generation), and fine-tuning are effective ways to improve LLM outputs. The authors recommend choosing the appropriate method based on the application context, task requirements, cost, and performance goals:

1. Start with Prompts: Begin new applications by refining prompts.
2. Use RAG for New Knowledge: Prefer RAG over fine-tuning when updating the model with new information.
3. Fine-Tuning for Specific Tasks: Consider fine-tuning only for tasks that prompt engineering cannot solve.

Prompt Engineering Insights

• Avoid the Ultimate Prompt Myth: Similar to software development, no single “ultimate prompt” solves all problems. Prompts should be concise and task-specific.
• Example Quantity: Use five or more examples in prompts. Too few can harm generalization, and too many can clutter the prompt.
• Reflect Input Distribution: Examples should mirror expected input diversity. For instance, in movie summaries, examples should represent various genres.

RAG Techniques

• Keyword Search: Don’t overlook traditional keyword search methods alongside embedding-based RAG.
• Document Quality: The quality of retrieved documents impacts the output. Relevant, concise, and detailed documents are preferred.
• Update with Ease: RAG allows for easy updates and fine-grained access control.

Fine-Tuning Advice

• Consider Costs: Fine-tuning is resource-intensive. If prompts cover most tasks, fine-tuning might not be necessary.
• Synthetic and Open Data: Use synthetic or open datasets to reduce annotation costs.

Agent and Workflow Strategies

• Clear Objectives: Define clear, specific tasks for agents, similar to managing junior employees.
• Deterministic Plans: Generate deterministic plans that can be tested and debugged.

Evaluation and Monitoring

• Unit Tests: Create unit tests from real input/output samples based on at least three practical metrics.
• Model as Judge: Use the most powerful model to compare outputs but allow for ties to avoid bias.

Additional Techniques

• Chain of Thought: Encourage the model to explain its reasoning before final answers to reduce hallucinations.
• Diverse Outputs: Achieve diversity not only by adjusting temperature but by altering prompt elements.

The shared experiences from these six authors provide valuable insights into the practical application of LLMs. Whether you’re a big tech engineer, an entrepreneur, or an independent developer, these tips can help optimize your use of large language models.

Follow and Explore More AI Insights

Follow us on Facebook: AI Insight Media.

Get updates on Twitter: AI Insight Media.

Explore AI INSIGHT MEDIA (AIM): www.aiinsightmedia.com.

Keywords

AI development, LLM applications, prompt engineering, RAG, fine-tuning, AI insights, practical AI tips