Prompt Engineering vs RAG for Editing Resumes

accomplishments and qualifications, I am seeing a lower yield of job application to interview, especially within the past year or so. In common with others, I have considered Large Language Models (LLMs) to assist with resume creation and editing. Ideally, you should create a new resume for every job you apply for, tailoring how you phrase your job experience to match the wording and peculiarities of the specific posting and company. If you are at least mid-career, you should have more work experience than can fit on a resume and will need to determine what to leave out of the resume. LLMs can help summarize, rephrase, and select the most relevant work experience to tailor a resume for a specific job posting. 

In this article, we will be looking at using prompt engineering and Retrieval-Augmented Generation (RAG) in Azure to supplement LLMs in writing a resume. LLMs can help write resumes without RAG, but using RAG allows us to experiment with RAG and determine if RAG results in better resumes. We will also compare a full LLM to a smaller language model. To compare the different cases, we use the following metrics (per Microsoft):

Groundedness: Groundedness evaluates how well the models answers align with information from the input source. LLMs should provide responses which are based on provided data. Any responses outside provided context are undesirable for writing a resume; we don’t want the LLM to make up work accomplishments!

Relevance: Relevance measures how pertinent model responses are to given questions. For our case, the LLM should provide resume content relevant to the given job description. 

Coherence: Coherence evaluates if provided language is clear, concise, and appropriate. This is especially important in resumes, where brevity and clarity are key.

Fluency: Fluency measures how well the LLM adheres to proper rules of English. Resume content should have proper grammar and spelling.

We will evaluate resume generation across three cases: 1) prompt engineering only, 2) RAG resume, and 3) RAG resume on a different base model. Evaluation will be qualitative according to the metrics above, with each scored from lower to higher as unsatisfactory, marginal, or satisfactory.

To have an LLM write the resume we must provide experience. This is mostly easily done with prompt engineering. Prompt engineering is a way of guiding LLMs to provide more helpful answers. LLMs are trained on very broad data sets (like the internet) to give them as much insight to human language and patterns as possible. However, this means they need context to provide specific (and helpful) responses. Prompt engineering techniques are ways of interacting with LLMs to improve their responses. 

To use prompt engineering to help write a resume, we provide the context (that the LLM is going to help us with a resume) and provide it work experience so it has data to draw from. Next, we provide the job posting and guide it through writing a resume. However, with a long job history we may encounter token limits, requiring either editing down experience or increasing the token limits (and thus cost) with the LLM and interface.

We will use Azure to conduct this experiment code-free. We begin with prompt engineering using the chat-gpt-4o foundation model. We will follow the Azure tutorial for creating a RAG-based app. To begin, follow all steps of “Create Foundry hub” and “Deploy Models” sections of the tutorial. For only using prompt engineering, skip the next few sections and go to “Text the Index in the Playground” and do the first two steps to deploy a chapt-gpt-4o foundation model. 

I will give as much detail on methods as possible, but I will not provide my own resume/work experience or provide the specific job posting I am using for privacy and professionalism concerns. 

We will mostly use the same prompts for each case. The prompt engineering case will add a step where we provide a master resume for the LLM to use as reference material. The prompts are adapted from a LinkedIn article on using prompts to write a resume with LLMs. To begin, we provide a system message in the “give the model instructions and context” box in the Azure playground. The system message is:

You are going to act as a professional resume writer skilled in presenting information concisely and using niche-appropriate language, while avoiding redundancy and cliché terms. Your task is to position my experience as a solution to my target job posting, tailoring it specifically so that it’s clear that I can manage the primary requirements of the job. It is important that you do not make up accomplishments I do not have. Only re-phrase and re-organize existing experience.

The system message provides the basic task to the LLM (write resumes) along with general guidelines around groundedness (do not make up accomplishments), relevance (position…to my target job posting), and coherence (avoiding redundancy and cliché terms). 

Now we provide a master resume. The prompt I used is: “First I will provide a list of my job accomplishments. Draw your resume bullets and summaries only from the information I give. Below is my work experience you are to use. Let me know if you understand.” I then provided a master resume. I will not use this prompt when using RAG.

Next, we give the LLM more context on the hiring company: “Here are some details about the hiring company so that you can more effectively tailor responses to match the company’s needs. Let me know if you understand.” I followed with the company information from the job posting; often job postings begin with a paragraph or two about the company. Then I provided the job posting set up with the following prompt:

Next, I will provide you the job description. We will use the job description to craft a resume with a summary statements and 3-5 summary bullets for each role I have, step by step. Let me know if you understand the job description, and please tell me the name of the role I am applying for. [paste job description here].

 To help target the LLM and provide more context for resume bullets, I next asked “What are the most important technical skills required for the job? Which technical skills would give me an advantage in this role?” and then “What are the most common areas of expertise for [the role]?”. The goal of these questions is to increase the relevance of provided bullets and summaries. 

Now it is time to start generating resume content. I had already decided on a rough format for the resume: begin with a paragraph summary, then provide 3-5 bullets for my two most recent jobs, and then 1-3 bullets for others. I conclude with an education section and summary of key relevant skills. The LLM will provide everything but the Education section. 

First, I asked it to provide a summary: 

Using my experience, write a 4-6 sentence summary, showing my 10+ years of experience, including only what’s relevant to the role. Showcase how my experience can address [X comapny’s] needs. Write it using passive third person, as appropriate for a resume. 

Now I ask it to provide bullet points for each of my jobs: 

Write 3-5 short bullet points from my experience at X company that best show how I fit the requirements for the role. Consider the technical requirements and most common experience discussed earlier. Start each bullet with an action verb, followed by the task, and conclude with the result. Where information is available, quantify each statement using numbers, percentages, and dollar amounts, etc. The bullet points should not exceed 2 lines on narrow margins using 10pt font. 

I repeat this prompt for each job, adding a clause to “Avoid redundancy with previous bullets” and changing the number of bullet points requested as described previously.

Finally, once I have bullets for each job experience, I ask the LLM to provide a summary of relevant technical skills:

Provide up to 5 concise bullet points summarizing key certifications and technical skills I have that are relevant to the role. Only provide a summary bullet if I have a specific relevant certification or technical skill.

The responses from these questions provide a starting point for a resume that should require minimal editing—mostly for format and editing out content that may be inaccurate or to ensure the resume fits on one page. So ends the prompt engineering case.

The next step beyond prompt engineering is RAG. RAG allows users to create their own libraries to serve as a knowledge base for LLMs to draw from. In this case, the document library consists of already created resumes. Older resumes provide more details on early career accomplishments. For more recent job experience, this approach is useful after you have already created a handful of resumes covering the spectrum of your work experience. Building a RAG from your resumes will help focus the LLM on your own experience base without needing a custom trained or tuned model. RAG isn’t necessary to using LLM to write a resume and will incur computational cost, but it could improve results as compared to only prompt engineering and make it easier to provide greater experience for the LLM to draw from.

We’ll use the same prompts for the RAG cases, except we’ll remove the first prompt providing work background, since the RAG will provide that. To use RAG, we return to the Azure tutorial, this time completing the “Add data to your project” and “Create an index for your data” sections. However, instead of using the data provided in the tutorial, create and upload a folder with all the resumes you would like the LLM to draw from. Once the indexing is complete, follow step four of “Test in the index in the playground” to add the data to the model’s context. After that, we repeat the prompts used earlier, except removing the first prompt providing work history.

Finally, to evaluate resume generation with a different foundational model, we deploy a new model to the project, this time chat-gpt-4o-mini, to evaluate its performance with RAG. LLMs have trillions of parameters, requiring enterprise level hosting. Small(er) language models (8 billion for chat-gpt-4o-mini vs 1.8 trillion for chat-gpt-4o) attempt to provide most of the capability of LLMs in a more compact and flexible form factor that supports localized deployment, especially important for data security and privacy of smaller companies that may not be able to support internal hosting of an LLM. Once the new model is deployed, we return to the playground, add the system message and data from the RAG, and repeat the same prompts as before.

The table below summarizes performance of each case:

The prompt engineering resume had substantial grounding issues such that I would not want to use it at all. It invented certifications I do not have and dollar amounts for improvements I did not make. The two RAG resumes were better, but still had some issues. Chat-gpt-4o was slightly more grounded, but still made some mistakes the mini did not. Considering the known issues of LLM hallucinations, we should expect to verify all statements. All three models were marginal on relevance; they did not include several important phrases from the job listing; they were able to phrase acceptable bullets, but these bullets could be improved by manual editing. The RAG resumes, especially on the full model, were slightly more relevant. All models were acceptable for coherence, though the RAG bullets were more concise. All models provided acceptable written English. If you have a large portfolio of resumes, it is probably worth using RAG if you want to generate resumes, if only to reduce the likelihood of hallucinations (that is, assuming you want to be honest—some of the accomplishments the LLM attributed to me were quite impressive!).

Some final thoughts on using LLMs to create resumes. The LLMs provided a good starting point for resumes, especially if you are finding it difficult to come up with new ideas or phrasing or want a new start to a resume. It is generally easier to edit a first draft resume than create a new one, so they can help job applicants in crafting resumes. However, I needed existing resume bullets and job experience available for the LLM to draw from. This means that I need to know how to write resume bullets. Writing these bullets is a perishable skill, so I recommend you not rely on LLMs to write all resumes, especially as you gain new work experience. Second, I needed to further reduce bullets and select which of the ones the LLM provided to keep; I could have avoided this by asking the LLM to write an entire one-page resume instead of proceeding step by step but this may have decreased quality (especially relevance) of the responses. Finally, I could have improved responses by using live interaction to help the LLM edit and improve its responses. However, I wanted to keep conditions as controlled as possible to improve comparison across the cases.

Using LLMs may be useful in the resume AI arms race to reduce time and effort for each individual resume, but remember to keep your skills sharp; the more you let something else do your thinking for you, the less capable you will be. Use LLMs to help edit and get new phrasing ideas for resumes, not to better flood job postings with applications. Most important, build human connections; a network and connections at a company is the best way to have your resume reviewed by the human eyes of a hiring manager rather than screened out by an HR bot.