A Generative AI Engineer has created a RAG application which can help employees retrieve answers from an internal knowledge base, such as Confluence pages or Google Drive. The prototype application is now working with some positive feedback from internal company testers. Now the Generative Al Engineer wants to formally evaluate the system??s performance and understand where to focus their efforts to further improve the system.
How should the Generative AI Engineer evaluate the system?

1. A. Use cosine similarity score to comprehensively evaluate the quality of the final generated answers.
2. B. Curate a dataset that can test the retrieval and generation components of the system separatel
3. C. Use MLflow??s built in evaluation metrics to perform the evaluation on the retrieval and generation components.
4. D. Benchmark multiple LLMs with the same data and pick the best LLM for the job.
5. E. Use an LLM-as-a-judge to evaluate the quality of the final answers generated.

Correct Answer: B

A Generative Al Engineer is tasked with improving the RAG quality by addressing its inflammatory outputs.
Which action would be most effective in mitigating the problem of offensive text outputs?

1. A. Increase the frequency of upstream data updates
2. B. Inform the user of the expected RAG behavior
3. C. Restrict access to the data sources to a limited number of users
4. D. Curate upstream data properly that includes manual review before it is fed into the RAG system

Correct Answer: D

A Generative Al Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author??s web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page
number, chapter number, book title), retrieved with the user??s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values.
Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)

1. A. Change embedding models and compare performance.
2. B. Add a classifier for user queries that predicts which book will best contain the answe
3. C. Use this to filter retrieval.
4. D. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapter
5. E. Choose the strategy that gives the best performance metric.
6. F. Pass known questions and best answers to an LLM and instruct the LLM to provide the best token coun
7. G. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.
8. H. Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chun
9. I. Optimize the chunking parameters based upon the values of the metric.

Correct Answer: CE

A Generative Al Engineer is ready to deploy an LLM application written using Foundation Model APIs. They want to follow security best practices for production scenarios
Which authentication method should they choose?

1. A. Use an access token belonging to service principals
2. B. Use a frequently rotated access token belonging to either a workspace user or a service principal
3. C. Use OAuth machine-to-machine authentication
4. D. Use an access token belonging to any workspace user

Correct Answer: A

After changing the response generating LLM in a RAG pipeline from GPT-4 to a model with a shorter context length that the company self-hosts, the Generative AI Engineer is getting the following error:

What TWO solutions should the Generative AI Engineer implement without changing the response generating model? (Choose two.)

1. A. Use a smaller embedding model to generate
2. B. Reduce the maximum output tokens of the new model
3. C. Decrease the chunk size of embedded documents
4. D. Reduce the number of records retrieved from the vector database
5. E. Retrain the response generating model using ALiBi

Correct Answer: CD