System Architecture

End-to-end view of the TPL LeadGen pipeline — auth, ingestion, CRM push, and RBAC

Browser
Pages — served from wwwroot/
login.html — Auto-redirect to Microsoft SSO · error display
index.html — Dashboard & stat cards
leads.html — Table · sort · search · CRM push with confirm dialog · “Sent” badges
worker-runs.html — Ingestion history · Run Now
settings.html — Schedule · limits · CRM credentials (per-permission)
users.html — User management · role assignment admin
roles.html — Role & permission editor admin
Vanilla JS · fetch() · no framework · no build step
js/auth.js — shared JWT module · requireAuth() · authFetch() · permission gates
Auth Module  js/auth.js
Captures ?sso_token= on every page load → stores in localStorage
Decodes JWT payload client-side to read user, role, permissions
requireAuth() — redirects unauthenticated users to /login.html
authFetch() — injects Authorization: Bearer <jwt> on all API calls
Hides / shows nav links based on admin.manage_users / admin.manage_roles
JWT stored in localStorage · expiry checked client-side · 401 response clears token & re-redirects
HTTPS
Bearer JWT
JSON
→
←
ASP.NET Core API  ·  net8.0
Static File Middleware
UseDefaultFiles() + UseStaticFiles()
Serves all 9 HTML pages & JS/CSS assets from wwwroot/
Auth enforced client-side via auth.js + server-side via [RequireAuthorization]
Auth & SSO Endpoints  public
GET /oauth2/login — redirect to Microsoft AAD /authorize
GET /oauth2/callback — exchange code → extract email → find/provision user → issue JWT → redirect
GET /api/auth/me — current user profile & permissions
TokenService issues HS256 JWT · claims: sub, email, name, role, roleId, permissions[] · 48 h expiry
Admin API  [auth + permission]
GET /api/users · POST · PUT /{id} · DELETE /{id} — admin.manage_users
PATCH /api/users/{id}/password — emergency password reset
GET /api/roles · POST · PUT /{id} · DELETE /{id} — admin.manage_roles
PUT /api/roles/{id}/permissions — replace permission set
GET /api/permissions — catalog grouped by area · GET /api/markets
Dashboard & Leads API  [auth + permission]
GET /api/leads — paginated · search · sort · includes CRM push flags · leads.view
PATCH /api/leads/{id}/status — leads.edit_status
PATCH /api/leads/{id}/notes — leads.edit_notes
GET /api/leads/stats — counts by status · dashboard.view
GET /api/worker-runs — incl. requestJson & responseJson · dashboard.view_worker_runs
GET /api/settings · POST — settings.view / settings.edit
POST /api/worker/run — queue manual trigger · settings.trigger_worker
POST /dev/ingest · POST /dev/skip-trace · /swagger dev only
CRM Sync API  user-triggered only
GET /api/leads/{id}/crm-sync — per-lead CRM status · leads.push_to_crm
POST /api/leads/{id}/crm-sync/ServiceMinder — push to ServiceMinder after confirm
POST /api/leads/{id}/crm-sync/ServiceTitan — push to ServiceTitan after confirm
GET /api/crm/settings · PUT — settings.crm.view / settings.crm.edit
Writes CrmPushLog per attempt (userId · userName · requestJson · externalId · timestamp) · Nothing runs in background · Confirmation dialog required before every push
Application Services  (Core layer)
TokenService — issues HS256 JWT with user claims + permissions JSON array
LeadIngestionService — dedup · batch create · ZIP rotation · stamps WorkerRunId on each lead · emits RentCastApiLog per ZIP
CrmSyncService — per-CRM push · writes CrmPushLog · updates CrmSyncStatus with externalId
SkipTraceService — enriches buyer contact data
WorkerSettingsService — live-editable schedule persisted to runtime-settings.json
EF Core 8 · Npgsql · IUnitOfWork pattern · ILeadRepository · ITargetZipCodeRepository
Background Worker  (IHostedService)
RentCastIngestionWorker — fires on startup, then every N hours
30-second poll loop — responds to schedule or “Run Now” trigger
Pre-generates WorkerRunId → passes to IngestAsync()
Persists WorkerRun record (requestJson + responseJson) + all RentCastApiLog rows per cycle
Max 264 API calls / run  ·  1,000 calls / month budget
EF Core
read / write
→
PostgreSQL  ·  tpl_leadgen
Auth & RBAC Tables
app_users  email (unique) · firstName · lastName · roleId · marketId · isActive
roles  name · description · isSystem — seeded: Admin, Viewer
permissions  key · name · groupId — 12 permission keys across 4 groups
role_permissions  roleId + permissionId (composite PK)
permission_groups  Dashboard · Leads · Settings · Admin
markets  DFW · Austin · Houston · San Antonio
Seeded at startup by AuthSeeder · Admin gets all 12 permissions · Viewer gets dashboard.view + leads.view
Lead Domain Tables
leads  status · source · sourceId (unique) · workerRunId (FK)
buyers  name · email · phone · skipTracedAt
properties  type · saleDate · price · sqft · beds/baths
addresses  street · city · state · zip · lat/lon
target_zip_codes  264 ZIPs · lastQueriedAt
outreach_events  channel · status · occurredAt
crm_sync_statuses  ServiceMinder · ServiceTitan · state · externalId
Audit & Debug Log Tables
crm_push_logs  leadId · userId · userName · crmType · requestId · requestJson · externalId · success · attemptedAt
rentcast_api_logs  workerRunId · zipCode · requestUrl · responseJson · statusCode · recordsReturned · success · attemptedAt
worker_runs  ranAt · succeeded · metrics · requestJson · responseJson
One CrmPushLog per push attempt (including failures) · One RentCastApiLog per ZIP per run · WorkerRun summarises the full cycle
EF Core Migrations
InitialSchema
AddPerformanceIndexes
AddWorkerRuns
AddAuthAndRbac
AddCrmPushLog
AddRentCastLogsAndWorkerRunDebug latest
Key indexes: leads.status · leads.sourceId (unique) · leads.workerRunId · app_users.email (unique) · worker_runs.ranAt · rentcast_api_logs.workerRunId
Npgsql.EntityFrameworkCore.PostgreSQL 8.0.10
HTTPS
(API only)
→
←
External Services
Microsoft Azure AD  SSO
login.microsoftonline.com/{tenantId}
oauth2/v2.0/authorize — OpenID Connect flow
oauth2/v2.0/token — exchange auth code for ID token
ID token claims used: email · given_name · family_name
New users auto-provisioned with Viewer role on first login
Configured via AzureAdOptions: TenantId · ClientId · ClientSecret · No passwords managed by app
RentCast API
api.rentcast.io  GET /v1/listings/sale
Recent deed sales by ZIP code · returns buyer name · sale price · address · bed/bath
Response JSON logged to rentcast_api_logs per call
Retry on 5xx / network only — 3 attempts, exponential back-off · 4xx returns failed result
1,000 calls / month budget  ·  ~264 calls per full scan · Stub mode when no API key
ServiceMinder CRM
REST API · runtime credentials from encrypted crm-settings.json
Returns confirmation / job ID stored as externalId in CrmSyncStatus
Purple “✓ Sent to Minder” badge shown on lead after successful push
Credentials encrypted at rest with AES-256-GCM · Push is user-triggered with confirmation · Singleton client (connection pool)
ServiceTitan CRM
OAuth 2.0 client credentials flow · access token cached in memory
Returns confirmation ID stored as externalId in CrmSyncStatus
Blue “✓ Sent to Titan” badge shown on lead after successful push
Credentials encrypted at rest with AES-256-GCM · Push is user-triggered with confirmation · Singleton client (token cache)
BatchSkipTracing API
Contact enrichment per buyer — email · phone · mailing address
Stamps SkipTracedAt on buyer after each attempt
On-demand via POST /dev/skip-trace  ·  future: scheduled cron
Dev vs Production — Key Differences
Aspect Local Development Azure Production
Environment Development Production
Config source appsettings.Development.Local.json (git-ignored) Azure App Settings — set via CLI, never in files
Authentication Microsoft AAD SSO — requires TenantId / ClientId / ClientSecret Same Microsoft AAD SSO — redirect URI must match App Registration
RentCast API key Usually not set → Stub mode — synthetic data Must be set → Live mode
Database Local PostgreSQL localhost:5432, no SSL Azure PostgreSQL Flexible Server, SslMode=Require
Dev endpoints POST /dev/ingest · POST /dev/skip-trace · Swagger at /swagger Not exposed — hidden when ASPNETCORE_ENVIRONMENT=Production
CRM credentials Edited via Settings UI → encrypted in crm-settings.json locally Same encrypted file — persists across deploys if using persistent storage
Logs Terminal console (colour-coded) az webapp log tail streaming
Auth Flow (SSO)
  1. Browser loads any protected page → no JWT → redirected to /login.html
  2. login.html auto-redirects to /oauth2/login (Microsoft account picker)
  3. User authenticates with Microsoft AAD
  4. /oauth2/callback exchanges auth code for ID token
  5. Email extracted from ID token → user looked up in DB
  6. Existing user: loaded with role + permissions → JWT issued
  7. New user: auto-created with Viewer role → admin assigns role later
  8. Redirect to /?sso_token=<jwt> → auth.js stores token, cleans URL
Ingestion Flow
  1. Worker fires on startup, then every N hours (configurable)
  2. Pre-generates a WorkerRunId GUID before calling IngestAsync
  3. Fetches the stalest target ZIPs up to API budget
  4. Calls RentCast per ZIP — logs raw response to RentCastApiLog
  5. Bulk dedup: one DB query returns already-ingested sourceIds
  6. Creates Lead → Property → Buyer → Address; stamps WorkerRunId on each lead
  7. Commits in one DB transaction; persists WorkerRun + all RentCastApiLog rows
CRM Push Flow
  1. User clicks “→ SM” or “→ ST” button in the Leads table
  2. Browser shows confirm dialog: “Send this lead to [CRM]? This cannot be undone.”
  3. On confirm → POST /api/leads/{id}/crm-sync/{crm}
  4. CrmSyncService pushes lead payload to the CRM API
  5. Confirmation ID stored in CrmSyncStatus.ExternalId
  6. CrmPushLog row written: userId, userName, requestJson, externalId, success/error, timestamp
  7. On success: button permanently replaced with colour-coded “✓ Sent” badge
  8. Subsequent page loads show badge instead of button (flag from crm_sync_statuses)
RBAC Flow
  1. JWT payload contains permissions[] JSON array baked in at login time
  2. Every API endpoint calls HasPermission(ctx, "key") before executing
  3. Frontend auth.js reads permissions from JWT to show/hide UI elements
  4. CRM settings split: settings.crm.view to see, settings.crm.edit to change
  5. Admin assigns roles via Users page → user re-logs in for new permissions to take effect
  6. Permission set per role managed via Roles page
Key Design Decisions
  • Clean architecture — Core layer has zero infrastructure dependencies
  • EF Core Fluent API only; no data annotations in Core entities
  • IUnitOfWork → same DbContext scope across all repositories
  • No background CRM sync — every CRM push is user-triggered with a confirm dialog
  • CRM credentials encrypted at rest (AES-256-GCM) — never stored in plain text
  • Every CRM push attempt written to crm_push_logs (success or failure)
  • Every RentCast API call logged to rentcast_api_logs with raw response JSON
  • Lead.WorkerRunId links each lead back to the exact ingestion run that created it
  • Microsoft SSO only — no passwords managed by the app
  • Stub HTTP clients in dev — no real API key needed to run locally
  • No frontend framework — vanilla JS, zero build step, zero npm