Reading Time: 16 minutes

EvidenceFit Studio: Engineering an Evidence-First AI Application for Job Seekers

From a frustrating job-search workflow to an AI-assisted SaaS platform built around real experience, defensible evidence, and better application decisions.

Project: EvidenceFit Studio
Platform: Web-based SaaS application
Technology: Python, PostgreSQL, REST APIs, Railway, large language models, AI voice services
Role: Founder, Product Owner, Technical Lead
Development approach: AI-assisted, human-directed
Status: Beta
Website: https://evidencefitstudio.com

Coming soon: https://resumefitstudio.com and applyd.fyi

1. The Problem: A Resume Is Not a Complete Career

EvidenceFit Studio grew out of a problem I was experiencing firsthand.

After spending much of my career in software engineering, customer support leadership, and customer operations, I was looking for my next leadership opportunity.

I began using AI tools to compare my resume against job descriptions, identify gaps, and tailor applications.

The results were useful, but I repeatedly encountered the same problems.

The tools often focused on matching keywords rather than understanding the experience behind them. They sometimes overlooked transferable skills, interpreted missing information as missing qualifications, or suggested stronger wording that was not adequately supported by the resume.

I also found myself repeating the same process for every application.

Upload a resume. Paste a job description. Analyze the match. Explain my experience. Rewrite sections. Check for exaggeration. Prepare for the interview.

Then do it again for the next opportunity.

The underlying problem was larger than resume writing.

A resume is a summary of someone’s career, not a complete record of everything they have accomplished.

If an employer asks for experience managing service delivery budgets, and the resume doesn’t explicitly mention budgeting, an AI system may conclude that the applicant lacks that experience.

But perhaps the applicant managed a multimillion-dollar operating budget and simply didn’t include it in that particular version of the resume.

An accurate system should recognize the uncertainty and ask.

That observation became the foundation of EvidenceFit Studio.

2. The Objective: Evidence Before Optimization

I wanted to create an application that helped job seekers make better decisions about the opportunities they pursued.

The objective was not to generate the most impressive-looking resume or maximize an ATS score at any cost.

It was to answer four questions:

1. What is the employer actually looking for?

2. Which requirements can the applicant demonstrate through real experience?

3. What important information is missing or unclear?

4. How can verified experience be presented more effectively without exaggeration?

The intended experience follows a structured sequence.

Understand the role → Identify the evidence → Resolve important gaps → Build the application → Prepare for the interview.

Users begin with their resume and a job description.

EvidenceFit extracts and evaluates the employer’s requirements, maps those requirements to the applicant’s experience, and distinguishes between direct support, transferable experience, uncertain information, and missing evidence.

When an important requirement cannot be established, the application can ask the user a targeted factual question rather than inventing an answer.

Verified information can then support more accurate tailoring and interview preparation.

My Design Principles

I established several guiding principles:

• Truth before optimization: Improving wording must not create qualifications or achievements that cannot be defended.

• Evidence before scoring: A numerical match should reflect underlying requirements and supporting evidence, not simply keyword overlap.

• Clarification before assumption: Missing information is not automatically missing experience.

• Human control: The applicant decides what is accurate and what is ultimately submitted.

• Explainability: Users should understand why their experience was classified as a match, partial match, or gap.

• Practical value: Analysis should lead to better application decisions, not simply more dashboards and metrics.

These principles shaped both the user experience and the technical implementation.

3. Ownership: What I Personally Contributed

EvidenceFit Studio was developed primarily through AI-assisted coding.

I used AI development tools to generate, modify, and troubleshoot substantial portions of the application.

My involvement was not limited to describing features and accepting whatever code was produced.

I was responsible for defining the product, structuring workflows, making architectural and operational decisions, identifying unexpected behaviour, evaluating proposed solutions, and deciding what was acceptable for deployment.

My previous background in software engineering helped me understand the relationships between application components, data, APIs, infrastructure, and security.

My experience leading technical support and customer operations influenced how I approached usability, incident investigation, documentation, testing, and ongoing supportability.

My Responsibilities

• Defined the product concept, user journeys, and evidence-first requirements.

• Established how job requirements and candidate evidence should be analyzed.

• Designed workflows for identifying, clarifying, and verifying experience.

• Directed AI-assisted implementation across frontend and backend functionality.

• Made decisions about hosting, persistence, environments, and third-party integrations.

• Evaluated scoring behaviour and investigated inconsistent results.

• Directed changes to authentication, security controls, and application reliability.

• Established staging and production workflows.

• Maintained development handoffs, issue tracking, and a master backlog.

• Conducted functional acceptance testing and evaluated the user experience.

• Controlled priorities, scope, and production acceptance decisions.

I remained accountable for the product’s behaviour even when AI generated the underlying implementation.

AI accelerated the creation of the software. It did not eliminate the responsibility to understand what the software was doing.

4. Architecture: Building an AI-Dependent SaaS Application

EvidenceFit Studio consists of a browser-based application, a server-side application layer, persistent storage, and integrations with external AI services.

Unlike a simple AI prompt interface, it needs to manage user accounts, resumes, job descriptions, application projects, verified career evidence, analysis results, and interview preparation.

These components need to work together while preserving the relationships between user-provided facts and AI-generated recommendations.

Application Interface

The web interface provides workflows for importing resumes, analyzing job descriptions, reviewing evidence, responding to clarification questions, tailoring application materials, and preparing for interviews.

An important design consideration was avoiding a single, overwhelming analysis screen.

The application needed to show users enough detail to make informed decisions without requiring them to interpret every intermediate AI output.

Backend and Application Logic

The backend coordinates application workflows, persistent records, user access, and external AI processing.

It must distinguish between information provided by the applicant, information extracted from source documents, and interpretations generated by AI.

That distinction is important because AI output should not automatically become an accepted fact about a person’s career.

PostgreSQL Database

EvidenceFit initially used SQLite and later migrated to PostgreSQL on Railway.

The migration reflected the application’s evolution toward a hosted, multi-user SaaS environment.

PostgreSQL provided a more appropriate foundation for shared persistent data, concurrent application use, and managed production and staging environments.

The migration also introduced additional operational responsibilities, including connection management, schema changes, backups, environment configuration, and deployment coordination.

External AI Services

EvidenceFit uses large language models for tasks that benefit from interpreting unstructured language.

These include examining job descriptions, understanding resume content, identifying possible evidence relationships, generating clarification questions, and supporting application preparation.

The interview roleplay functionality introduces additional services for voice-related capabilities.

Keeping those integrations behind the application layer allows provider configuration, request handling, and usage controls to be managed separately from the browser interface.

Production and Staging Environments

As the application became more complex, I established a separate staging environment on Railway.

The objective was to evaluate changes without using production as the primary testing environment.

Staging introduced its own challenges involving OAuth configuration, environment variables, allowed origins, and database separation.

It also created a clearer distinction between generating a code change, deploying that change, and accepting its actual behaviour.

5. Technical Decisions and Trade-offs

Decision 1: Build Around Evidence, Not Just Prompts

A conventional AI resume tool can send a resume and job description to a model and request an improved document.

That approach is fast, but it leaves substantial control over factual interpretation to the model.

I wanted a more structured process.

EvidenceFit separates job requirements, resume evidence, user-confirmed facts, and generated application wording.

The advantage is greater traceability and more opportunities to prevent unsupported claims.

The trade-off is complexity.

Instead of one prompt and one response, the application must coordinate multiple analysis and verification steps while maintaining consistency between them.

Decision 2: Ask Questions Rather Than Invent Answers

If a job requires experience with a particular responsibility and the resume does not establish that experience, the system can ask the applicant.

For example, a resume may demonstrate customer support leadership but not explicitly mention financial responsibility.

Rather than creating a statement about budget ownership, EvidenceFit can ask whether the applicant has managed operating budgets and request supporting details.

The user can confirm the experience, provide factual context, or indicate that the requirement is not supported.

This creates an additional interaction, but it also improves the integrity of the evidence used later.

A faster workflow would be to let the AI make assumptions.

I deliberately rejected that as the default behaviour.

Decision 3: Use Scoring as Decision Support

A match score can help someone quickly understand how their experience relates to a position.

However, a score is only useful if the underlying requirement extraction and evidence classification are meaningful.

I treated scoring as a decision-support mechanism rather than a prediction that someone would be interviewed or hired.

This became especially important when testing exposed inconsistent results.

Decision 4: Separate Staging From Production

Working rapidly with AI-generated code increases the risk of introducing regressions.

Changes to one workflow can affect shared data, authentication, or analysis behaviour elsewhere.

A staging environment allows changes to be reviewed before production deployment.

The trade-off is additional operational complexity and infrastructure configuration.

For a growing SaaS application, I considered that worthwhile.

Decision 5: Keep Interview Preparation Connected to Evidence

Interview practice is often treated as a separate activity from resume preparation.

I wanted EvidenceFit to use the same verified background and job requirements when helping applicants prepare.

This reduces the need to repeatedly explain the same career history.

It also supports a more coherent experience: the claims presented in an application should be claims the applicant can discuss and defend in an interview.

6. Engineering Challenge: When the Same Inputs Produce Different Scores

One of the most instructive problems involved scoring consistency.

During testing, I found that the same resume and job description could produce materially different match scores across repeated analyses.

In one set of tests, the results included 73%, 53%, 68%, and 52%.

The variability created a serious product problem.

A user making an application decision should not receive substantially different guidance simply because the system ran the analysis again.

Investigating the Problem

The problem was not necessarily a faulty arithmetic calculation.

Language models are probabilistic systems. They can interpret, group, and classify the same text differently across separate requests.

If the extracted requirements or evidence classifications change, the resulting scores can also change.

The issue therefore needed to be examined upstream, not merely at the final scoring display.

The Design Response

One measure introduced was to associate analyses with normalized inputs and an analysis-version fingerprint.

Where the relevant inputs and processing version were unchanged, the application could reuse an existing result instead of unnecessarily requesting a fresh analysis.

This improved repeatability for equivalent requests and avoided redundant processing.

It also had potential cost and performance benefits.

What This Did Not Solve

Reusing an analysis does not prove that the analysis is correct.

A consistently reproduced score can still be inaccurate if the original extraction missed important requirements or misclassified evidence.

That distinction became a central engineering lesson.

Consistency and correctness are two different quality attributes.

The longer-term solution requires stronger evaluation of requirement coverage, evidence classification, and how the system determines the importance of individual requirements.

7. Engineering Challenge: More Accurate Analysis Can Produce a Lower Score

Another valuable example occurred while evaluating a real job description for an Ericsson customer operations role.

Earlier production analyses produced scores in the 80% range.

A new analysis in staging produced a score of approximately 55%.

Initially, the lower result appeared suspicious.

But closer examination showed that staging had identified more job requirements and highlighted more areas requiring verification.

The production analysis had identified roughly 10 requirements in one run, while the staging analysis identified approximately 22 high-priority requirements.

The staging workflow also generated clarification questions for experience that was not adequately established.

When I reviewed those questions, I could not truthfully answer yes to many of them.

That changed my interpretation of the results.

The higher score was not necessarily more accurate simply because it looked better.

The lower score might have been more defensible because the application was examining requirements that an earlier analysis had overlooked.

The Engineering Lesson

This exposed a difficult product-design question:

How do you prevent improvements in requirement extraction from making the application appear inconsistent or unreliable?

It also reinforced the need to evaluate the full analysis pipeline rather than treating the final score as the only meaningful output.

The investigation remained ongoing, but it was a useful example of why I wanted EvidenceFit to prioritize evidence over flattering results.

8. Engineering Challenge: A Staging Failure Wasn’t Necessarily a Code Defect

Separating staging from production introduced a different class of problems.

During an early staging analysis, a request failed because the staging origin had not been correctly included in the backend’s allowed-origin and CSRF configuration.

The visible symptom was a failed application operation.

However, the underlying issue was environment configuration rather than the analysis logic itself.

This distinction matters.

A browser request can fail because of application defects, network problems, authentication, CSRF protection, configuration differences, or service availability.

Correct diagnosis requires identifying which layer is responsible before changing code.

The staging origin configuration was corrected, allowing the analysis workflow to proceed.

The experience reinforced the importance of environment-specific configuration and validating complete workflows after deployment.

9. Security: Protecting Accounts, Evidence, and AI Endpoints

EvidenceFit handles resumes, career histories, job descriptions, and potentially sensitive information about professional experience.

It also integrates with external AI services that introduce cost, reliability, and data-handling considerations.

Security therefore needed to be part of the implementation rather than something addressed only after launch.

Authentication and Authorization

The application supports account-based access, including Google OAuth.

Authentication establishes who is using the application.

Authorization is the separate responsibility of ensuring users can access only the resources and operations permitted to them.

This is particularly important for an application that stores multiple users’ resumes, career evidence, and application projects.

Cross-Site Request Forgery Protection

CSRF protection helps prevent unauthorized state-changing requests that misuse an authenticated browser session.

The staging-origin investigation demonstrated that these protections also need to be configured correctly for each environment.

A security control that rejects legitimate traffic because of configuration errors creates an operational problem, while an overly permissive configuration can introduce risk.

Rate Limiting and Resource Abuse

EvidenceFit exposes a free resume-analysis tool on its public website.

Public AI-powered endpoints are attractive targets for automated abuse because requests may consume paid AI services.

Rate limiting and backend request validation help control that exposure.

The objective is not simply to protect availability, but also to prevent unexpected costs.

Server-Side Secret Management

API credentials and sensitive configuration belong on the server side rather than being exposed to the browser.

The application architecture separates external-service integration from the publicly delivered interface so that credentials do not need to be embedded in client-side code.

Prompt Injection and Unsupported AI Output

The application must also treat resumes and job descriptions as untrusted source content.

A job description is information to analyze, not a source of operational instructions that should override the application’s rules.

Similarly, generated recommendations must remain subordinate to the candidate’s verified evidence.

This requires controls beyond simply instructing an AI model not to fabricate information.

Prompt and output validation, clear trust boundaries, and user review are continuing engineering concerns.

Security as an Ongoing Process

I do not consider implementing a collection of controls equivalent to completing a security audit.

Security requires continual testing, dependency maintenance, configuration review, and reassessment as the application changes.

For an AI-assisted codebase, that discipline is especially important because generated implementations can introduce vulnerabilities that are not obvious during ordinary functional testing.

10. Testing: Verifying Behaviour, Not Just Successful Deployment

A successful deployment does not establish that a feature works correctly.

EvidenceFit’s development process has included hands-on workflow testing, regression investigation, comparison of analysis results, and verification of changes across environments.

Functional Acceptance Testing

I have used real resumes and job descriptions to evaluate the application’s ability to identify requirements, classify evidence, ask clarification questions, and produce usable application materials.

Testing with genuine career information has been valuable because it exposes ambiguity that simplified examples often miss.

Regression Investigation

Changes to requirement extraction, evidence baselines, or scoring logic can affect multiple downstream workflows.

For example, a change intended to identify more requirements may also alter the apparent match score, the number of clarification questions, and the evidence available for resume tailoring.

Testing therefore needs to examine the relationships between those outputs rather than treating each feature in isolation.

Production and Staging Verification

Staging provides a controlled environment for reviewing changes before they reach production.

I retain responsibility for personally verifying live application behaviour rather than accepting a successful build or deployment message as proof of correctness.

Areas Requiring Further Improvement

The application would benefit from stronger automated regression fixtures, repeatable test datasets, and defined acceptance thresholds for AI analysis quality.

In particular, testing should help answer:

• Did the system identify the material responsibilities and qualifications in the job description?

• Did it classify the applicant’s evidence accurately?

• Did it distinguish missing information from missing experience?

• Did any generated claim exceed what the evidence supports?

• Did the change introduce unexpected differences elsewhere?

These are more meaningful tests than checking only whether an API returned a successful response.

11. Maintainability: Managing a Codebase Built With AI

One of the less visible challenges of AI-assisted development is maintaining continuity across development sessions.

An AI coding assistant may understand the immediate task while lacking important context about earlier decisions, known defects, or the intended architectural direction.

Without structure, it becomes easy to recreate old problems or introduce changes that conflict with previous decisions.

I established several working practices to reduce that risk.

Development Handoffs

At the end of development sessions, I maintain a handoff document describing the current state of the application.

It includes completed work, unresolved issues, relevant environment state, and the next recommended actions.

The purpose is to establish continuity without relying entirely on conversational memory.

Master Backlog

I maintain a structured backlog of defects, improvements, and future capabilities.

Work items use identifiers that can be connected to implementation discussions and repository changes.

This helps distinguish committed work from suggestions or ideas that have not been approved.

Controlled Scope

One of my recurring challenges has been recognizing when an attractive feature is not yet justified.

AI makes feature implementation feel inexpensive, but every added feature introduces potential testing, documentation, security, and maintenance obligations.

I increasingly apply three questions before authorizing work:

1. What user problem does this solve?

2. What evidence suggests it matters now?

3. What other work would be delayed if I built it?

This has become an important part of managing the product’s complexity.

Database Change Discipline

Persistent data requires particular care.

My development workflow separates code changes from direct database administration.

Production and staging database queries, exports, and sensitive database operations remain under my control rather than being delegated to the AI development assistant.

This reduces the risk of an automated coding session unexpectedly modifying live data.

It also means database changes require deliberate coordination between implementation and deployment.

12. Interview Roleplay: Extending the Application Beyond Resume Generation

I wanted EvidenceFit to support more than the written application.

A resume may establish that someone has led a team, managed a budget, or delivered a technical program.

An interview requires the candidate to explain those experiences clearly and respond to questions from different perspectives.

EvidenceFit includes interview preparation and roleplay workflows designed around the applicant’s background and the target position.

These include different interviewer perspectives, such as recruiters, hiring managers, technical and operations interviewers, and skeptical senior leaders.

Why This Was Technically More Difficult

A text-based resume analysis is primarily an asynchronous processing problem.

Voice-based interview practice adds different engineering requirements:

• Capturing and processing spoken responses.

• Generating appropriate follow-up questions.

• Managing conversational context.

• Handling latency and external service failures.

• Controlling service costs.

• Providing useful feedback after the interview.

The experience depends on responsiveness as well as content accuracy.

A technically correct response that arrives too slowly can make a conversation feel unnatural.

Provider Reliability and Fallbacks

I explored a primary voice-generation service with a fallback provider.

The intention was to avoid making the entire interview experience dependent on one service remaining consistently responsive.

This introduced additional complexity involving timeouts, failure handling, configuration, and cost.

The broader lesson was that integrating an AI service is not simply about connecting to an API.

The application must also behave sensibly when that service is slow, unavailable, or returns an unexpected result.

13. Outcome: From a Personal Workflow to a Working SaaS Beta

EvidenceFit Studio has progressed from a collection of manual AI-assisted job-search activities into a working web-based application with integrated workflows.

The platform brings together job requirement analysis, evidence mapping, targeted clarification, resume tailoring, application support, and interview preparation.

It includes account-based access, persistent application projects, a backend database, external AI integrations, and separate production and staging environments.

The public-facing website also offers a free resume-check experience.

The application is being introduced through a beta process rather than presented as a finished, mature enterprise product.

At this stage, I have demonstrated that the core product workflow can be implemented and used.

However, that does not establish that the matching methodology predicts hiring outcomes or that the application has achieved sustained commercial adoption.

Those are questions that require additional usage, measurement, and validation.

The product’s progress is meaningful, but the distinction between a working beta and a proven business matters.

14. What I Learned

AI Can Accelerate Development Without Guaranteeing Quality

AI-assisted coding allowed me to build and connect functionality across several technical domains.

It also made it possible to introduce complexity much faster than traditional development approaches might have allowed.

The challenge shifted toward evaluating the quality of implementation, identifying regressions, managing dependencies, and retaining a coherent architectural direction.

A More Impressive Score Is Not Necessarily a More Accurate Score

The scoring investigations taught me to question outputs that looked good but were not adequately supported.

This principle applies to AI products generally.

A convincing answer is not the same as a correct answer.

Product and Engineering Decisions Are Connected

The decision to ask applicants for clarification was fundamentally a product decision.

But it also affected data models, workflow states, persistence, prompt design, and downstream generation.

Similarly, the decision to introduce staging was operational, but required changes to configuration, authentication, deployment, and testing.

Documentation Reduces Operational Risk

Development handoffs, a structured backlog, and explicit next steps have helped maintain continuity across AI-assisted development sessions.

These practices do not replace good code design or automated tests, but they reduce the risk of losing context and repeating decisions.

The Best Feature Is Sometimes the One You Don’t Build

I have had to challenge my own tendency to add capabilities because they seem valuable or because competing applications offer them.

The more important question is whether a feature solves a demonstrated user problem.

That discipline is becoming as important as implementation speed.

15. What I Would Improve Next

As EvidenceFit Studio matures, my priorities would include:

Stronger automated regression coverage. Establish known input datasets with expected requirement extraction, evidence classifications, and supported claims.

Better analysis-quality measurement. Separate scoring repeatability from scoring correctness, with clearer evaluation criteria.

More robust security verification. Expand systematic testing of authentication, authorization, input boundaries, public endpoints, and cross-account data isolation.

Improved operational visibility. Make it easier to diagnose processing failures, external-service latency, and unexpected costs without exposing unnecessary personal information.

More disciplined feature prioritization. Focus development on reliability and measurable user value before introducing additional workflows.

User outcome validation. Evaluate whether the product genuinely helps users identify suitable opportunities, reduce application effort, and prepare more effectively for interviews.

The objective is to make the product increasingly reliable, explainable, and maintainable, not simply larger.

16. Closing Thoughts

EvidenceFit Studio began with a practical frustration.

I wanted a better way to understand whether my experience matched the opportunities I was considering, and to communicate that experience without exaggeration.

Building the application has given me an opportunity to combine my background in software engineering, technical operations, customer support leadership, and product development.

It has also reinforced the limitations of AI-assisted development.

Generating code is not the same as engineering a reliable system.

Generating a resume is not the same as representing a person’s experience accurately.

And generating a score is not the same as helping someone make a better decision.

The quality of an AI application depends not only on what it produces, but on how its outputs are evaluated, verified, explained, and controlled.

That is the principle guiding EvidenceFit Studio.

My goal is to build a product that helps job seekers recognize the value of their actual experience, understand where they genuinely fit, and present themselves with greater clarity and confidence.

Not by inventing a better career.

By making the career they already have easier to understand.

Explore EvidenceFit Studio: https://evidencefitstudio.com

Try the free resume check: https://evidencefitstudio.com/resume-check/

Acknowledgements

I’ve also been fortunate to have two outstanding people contribute their experience and perspectives to the development of EvidenceFit Studio.

Robert J. Benjamin is a former Senior Vice President of Sales with more than 25 years of experience building and leading sales organizations. His extensive background in reviewing resumes, interviewing candidates, evaluating talent, and building sustainable, high-performing teams has provided valuable insight into how hiring decisions are made.

Eric Park is a CEO and former CTO who leads his own boutique technology consulting firm. His experience spanning software engineering, technical leadership, enterprise architecture, and business strategy has brought an important technical and operational perspective to the project.

I’ve also had the opportunity to speak with numerous HR managers, directors, and practitioners throughout the development of EvidenceFit Studio. These conversations have provided valuable perspectives on recruitment practices, candidate evaluation, resume screening, and the challenges faced by both job seekers and hiring teams.

Together, these perspectives have helped me challenge assumptions, refine the product’s direction, and better understand how EvidenceFit Studio can address real-world problems rather than simply adding features.

I’m grateful to everyone who has generously shared their time, experience, and expertise.