Secure every S3-backed document retrieval path in the Django API, Temporal document pipeline, and intra-frontend. The rollout first makes each document bucket private behind a CloudFront distribution and removes direct browser S3 access; it then makes each browser retrieval backend-authorized and short-lived. Internal processing continues to use server IAM/S3 APIs and stable object references, so workflow duration cannot invalidate a reference needed for OCR, masking, merging, password unlock, or Nucleus upload.
document_upload.services.DocumentS3Service, documents.services.s3.DocumentS3Service, PraveshCAS.utils.s3_utils, Account Aggregator CRIF direct boto3, BSA direct/bucket-override helpers, connector and Sahyogi wrappers, and app-local clients (field visits, Exotel, WhatsApp). Several emit raw virtual-hosted S3 URLs.lead_collection.models.Document.document_url, a mandatory raw URL. Both document_upload services return raw S3 HTTPS URLs after put/copy, and lead_collection persists those values. documents.services.public_urls can derive a CDN URL from an S3 key or stored S3 URL, but it has no signing or authorization.PraveshCAS.utils.s3_utils.s3_url_to_base64() parses S3 URLs and performs an authenticated GetObject, falling back to unauthenticated HTTP. lead_collection Nucleus callbacks currently pass document_url references into the Nucleus attachment path.intra-frontend/components/branch-iq/PreLoginApplications.tsx directly embeds API-returned document_url values in image and iframe viewers. A valid signed URL can support that rendering, but it must be obtained after each authenticated view action rather than embedded in application-list payloads.saarathi_sahyogi, banking, and /api/docs/ configuration already select CloudFront prefixes. These are unauthenticated prefix rewrites, so they must not be treated as an authorization mechanism.lead_collection: BranchIQ returns Document.document_url and CreditReport.pdf_report_url; /api/docs/, StateManager, and collateral APIs publish CDN-rewritten URLs; Partners, Sahyogi, Collect IQ, BSA/AA, credit reports, and several older frontend surfaces also return or open object URLs.| Surface | Current object/reference behavior | Required cutover | Delivery class |
|---|---|---|---|
lead_collection + document_upload |
Uploads temp objects, copies/merges them, and persists raw Document.document_url. |
Preserve temp/permanent keys for workers; return a document resource ID from client APIs; remove raw URL serialization. | Mutable pipeline |
temporal_workflows |
Orchestrates temp reference IDs plus persisted URLs for OCR, masking, BSA password unlock, PDF/Excel merge, and Nucleus preparation. | Carry bucket/key locators only; all reads/copies use worker/Django IAM; never sign inside a workflow. |
Internal processing |
PraveshCAS |
Parses S3 URLs, uses boto3, then falls back to HTTP before base64 attachment. | Normalize locators; remove HTTP fallback; attach Nucleus payloads from authenticated reads. | Internal processing |
branchiq + BranchIQ frontend |
Pre-Login detail and bureau endpoints return direct URLs; document and bureau views have no visible permission declaration; frontend embeds those URLs. | Enforce branch/state authorization for list, document, bureau, and access endpoints; use per-document on-click access URLs. | Read-only PII |
documents, StateManager, collateral, disbursal_ops |
ApplicationDocument.s3_document_url is exposed as an unauthenticated CDN rewrite and is copied into linked/collateral responses. |
Add resource access endpoint and replace all hydrated URL fields with IDs/metadata; retain existing application permission service. | Read-only then mutable |
credit_reports, credit_iq, customer_profile |
Reports are uploaded to S3; Credit IQ mixes CDN URLs, S3 presigns, and an optional same-origin PDF stream. | Sign through the shared CloudFront signer, use the signed CloudFront URL directly in the iframe/new tab, and remove both direct S3 presign and server-stream fallbacks. | Read-only PII |
bsa, banking, account_aggregator |
Bank statements, FI PDFs, and analytics spreadsheets are persisted as public URLs and later merged/attached. BSA endpoints shown are AllowAny. |
Authenticate/authorize consumer endpoints, use stable locators for processing, then issue download access per analysis/document resource. | Mutable pipeline |
Partners + saarathi_sahyogi |
App-specific wrappers upload PAN/Aadhaar/Sahyogi artifacts and return/store raw URL values; Sahyogi rewrites host to CDN. | Reuse shared locator/signer and per-prospect or Sahyogi document authorization; remove host-rewrite-as-access behavior. | Read-only PII |
collect_iq |
Field-visit service generates CDN photo URLs consumed by timeline UI. | Map field-visit keys to access resources scoped to the collection case; sign only on photo preview. | Read-only PII |
doc_gen, loan_origination_system, disbursal_ops |
Generated agreement/CAM/form assets resolve S3/CDN references server-side and surface document URLs in agreement views. | Preserve IAM asset reads; attach generated files to document resources before any browser retrieval. | Internal + read-only |
exotel_calling, whatsapp_comms |
App-local boto3 clients manage recordings/media. | Inventory model/API consumers, move PII recordings/media to resource-authorized access; do not merge their buckets with loan-doc origins. | Read-only PII |
| Marketing, APK/tutorial, general-information frontend surfaces | Object links may be intentionally distributable rather than customer PII. | Classify each prefix as public static or protected before changing its behavior; public static content uses a distinct CloudFront behavior and bucket/prefix. | Public/static or protected |
intra-frontend broadly |
BranchIQ, due diligence, deviations, property, collect-IQ, agreement, dashboard, tutorial, and legacy frontend-next components open URLs directly. |
Replace each endpoint contract first, then change the named UI handler to request a capability on click; remove URL values from TypeScript domain models. | Browser consumers |
The inventory search found direct boto3/S3 wrapper use in the rows above; apps not represented in these results are not assumed safe. The shared service must emit a metric for every rejected/unmapped locator, and release cannot finish until the metric is zero for active document traffic.
The hardening boundary is a backend-authorized document resource, not a CDN path. CloudFront signed URLs are selected for browser GETs because they retain CDN range/stream performance while authorization stays in Django. Signed cookies would grant a broader path/session capability than a single view/download action; backend streaming imposes Django bandwidth and range-request load; S3 presigned GETs bypass CloudFront policy and cache controls. Direct S3 upload is deliberately separate: use presigned POST only after retrieval is enforced and the backend owns the final object association.
Browser document bytes never traverse Django or a Lambda. Django’s access endpoint performs only low-cost identity/scope checks and returns a signed CloudFront URL; the frontend or mobile client uses that URL directly against CloudFront. This preserves CDN range requests and avoids application-worker bandwidth, response buffering, and file-size scaling.
If compliance requires the CloudFront private signing key to be isolated from Django, deploy an internal-only signer Lambda behind IAM-authenticated API Gateway. Its exact request is {bucket, key, expires_at, disposition, request_id} from Django after authorization; it validates the configured bucket/prefix mapping and returns {url, expires_at}. It has no browser endpoint, database access, or document-byte path. Django remains the authorization authority because it owns application, customer, and branch/state policy; the Lambda is only a key-isolation/signing implementation.
Goal 1 in plain language: today a person can open https://bucket.s3.ap-south-1.amazonaws.com/APP123/pan.jpg directly if the bucket is public. After Goal 1, that exact S3 link must return 403 Access Denied. The same file is fetched by CloudFront, for example https://assets.example.com/APP123/pan.jpg; at this stage that CDN URL is intentionally still open to anyone who knows the path. This is the compliance/private-bucket step, not the PII access-control step.
AWS terms used above:
GetObject, PutObject, CopyObject, or HeadObject permissions their processing path needs./loan-documents/* → primary document bucket or /field-visits/* → field-visit bucket. Keep unrelated public marketing assets in another behavior/prefix.https://intra.example.com to render a PDF iframe, but not an unknown website.Expires, Signature, and Key-Pair-Id for each request but caches one copy of APP123/pan.jpg; it does not create a separate cache object for every employee’s signed URL.Record a simple mapping in deployment configuration: loan-documents → d123.cloudfront.net → primary-bucket, field-visits → d456.cloudfront.net → field-visit-bucket, plus each distribution’s signing key-group ID. The app uses this map to turn a stored object key into a direct client URL; it never exposes the bucket hostname.
Establish an inventory-backed storage policy before changing object permissions. Create a Django secure_storage service that normalizes only configured S3 URL styles (https://bucket.s3.region.amazonaws.com/key, path-style HTTPS, s3://bucket/key, and bare key for the explicitly supplied bucket) to the immutable pair (bucket, key). It must reject unconfigured buckets, empty keys, CDN URLs without a configured distribution mapping, and arbitrary HTTP URLs. Reuse it in the existing document_upload, documents, PraveshCAS, BSA, CRIF, connector, Sahyogi, Collect IQ, credit-report, and document-generation integrations; retire their local public-URL constructors and URL splitters. Store existing raw S3 URLs unchanged for this rollout, but treat them as server-only object locators and never as browser output.
Implement Goal 1 at AWS and service boundaries. For every document-bearing bucket discovered by the inventory—primary S3_BUCKET_NAME, BSA, AA/CRIF, connector, Sahyogi, field-visit, credit-report, temporary workflow, Exotel recordings, WhatsApp media, generated-documents, and any environment-specific equivalents—enable S3 Block Public Access, Bucket owner enforced object ownership, default SSE-KMS, and a bucket policy allowing s3:GetObject only from its corresponding CloudFront Origin Access Control plus least-privilege backend/worker IAM roles. Configure CloudFront behaviors per bucket/prefix with HTTPS-only viewers, no directory listing, key-preserving origin paths, a CORS response policy permitting the approved browser origins, and cache keys that exclude CloudFront signature parameters. During this phase, use the existing CDN host mapping only for browser-facing response serialization; raw S3 URLs remain valid only to trusted SDK clients. The infrastructure is not represented in the scanned deployment directories, so implement this exact configuration in the organization’s AWS/IaC repository and record the distribution ID, domain, key group ID, and bucket-prefix mapping in environment configuration.
Separate stable internal references from browser capabilities. Replace every processing boundary that currently calls a URL a “reference” with the normalized object locator: Temporal upload activities continue to pass the temporary object key/reference ID; Django copy, OCR, Aadhaar masking, PDF/Excel merge, BSA unlock, collateral PDF creation, and Nucleus base64 conversion resolve that locator through authenticated S3 GetObject/CopyObject. Remove requests.get() fallbacks from PraveshCAS.utils.s3_utils.s3_url_to_base64 and document_upload.services.PdfMergeService.merge_pdf_urls; these fallbacks would fail after private-bucket cutover and permit arbitrary-URL fetching. Permit an external source only in a dedicated ingestion path that downloads it once under an explicit allowlist and writes it to a controlled temporary object. PraveshCAS continues attaching base64 to Nucleus from IAM-backed object reads; it must never consume a CloudFront-signed browser URL.
Implement Goal 2 with CloudFront signed URLs, not S3 presigned URLs or server byte proxying. Add a shared signer that takes an authorized (bucket, key), maps it to the configured CloudFront distribution path, and issues a CloudFront canned-policy URL with a five-minute expiry. Configure its private key as a secret manager-injected PEM and identify the CloudFront key group/public key through environment settings; fail closed with a 503 DOCUMENT_ACCESS_UNAVAILABLE when signing configuration is absent. Do not place signed URLs in models, Temporal workflow state, logs, queues, notifications, or cached list responses. The signer returns exactly {url, expires_at} and emits an audit event with actor, application/document identifier, action (view or download), and expiry, never the signed query string.
Harden browser retrieval by resource, not path. Add an authenticated access endpoint beside each document-owning API that accepts only its database document identifier and an optional disposition=inline|attachment; it resolves the object key server-side, checks the existing application/customer/branch/state authorization before signing, and returns the shared {url, expires_at} capability. For the documents app, add the endpoint under applications/<application_id>/documents/<document_id>/access/ and reuse DocumentLookupMixin plus the application permission checks in DocumentBaseView. For BranchIQ Pre-Login, make ApplicationDocumentsView return document_id, metadata, and no document_url; add GET pre-login/documents/<app_id>/<document_id>/access/, require IsAuthenticated and BranchIQPermission, and apply the same geo_query_filter_access_error scope used by LoanApplicationListView before verifying that the document belongs to a participant of app_id. Apply the same response cutover to BranchIQ bureau reports, Partners connector documents, Sahyogi, StateManager hydration, collateral/due-diligence/disbursal document views, Collect IQ photos, and credit-report viewing endpoints. Replace raw/CDN URL fields in their JSON contracts with stable resource IDs and metadata; no compatibility URL field remains.
Change intra-frontend viewers to request access only on user intent. In lib/api/pre-login.ts, model the document list item as {document_id, document_type, name, verification_status, is_img, is_pdf} and add getDocumentAccess(appId, documentId, disposition?). In components/branch-iq/PreLoginApplications.tsx, retain document metadata in state, call that API only when View is clicked, and set the returned signed URL in the image/iframe modal. On expiry or 403, clear the stale URL, obtain one fresh signed URL through the access endpoint, and retry once; never retain the URL in local storage or a query string. Apply this interaction model to every frontend component identified in the inventory that presently opens document_url, s3_document_url, file_url, report_url, or a direct S3/CDN URL.
Roll out by access class, keeping downstream work unblocked. First convert immutable, read-only PII assets—PAN, masked Aadhaar, property documents, applicant photos, credit reports, and generated PDFs—to the access endpoint contract. Next convert mutable/read-download-update flows—bank statements, BSA/AA outputs, collateral image sets, merged PDFs/Excels, and document-manager uploads—after their Temporal/Django transitions use internal locators exclusively. Finally convert direct-to-S3 upload flows to a separate presigned-POST initiation/complete contract: backend authorizes the target prefix and content constraints, issues a five-minute generate_presigned_post, then validates HeadObject and persists the stable locator on completion. Do not issue a direct upload capability before retrieval authorization and internal-reference cutover are live.
Add secure_storage.StorageLocator as the canonical server-side index without changing or deleting existing objects. Its fields are owner_content_type, owner_pk, owner_field, bucket, object_key, source_value, state (MAPPED, MISSING, INVALID, AMBIGUOUS), and timestamps; (owner_content_type, owner_pk, owner_field) is unique. source_value is retained only for audit and rollback. The existing URL columns remain internal locators during the migration, but no serializer may expose them.
Add one declarative owner map in secure_storage rather than scattering migration code: each entry names the Django model, URL field, fallback bucket (if the old value is a bare key), document resource resolver, and access policy. Seed entries for lead_collection.Document.document_url, documents.ApplicationDocument.s3_document_url, credit_reports.CreditReport.pdf_report_url, collateral.CollateralImage.file_url, BSA/AA result URL fields, Partners prospect documents, Sahyogi documents, Collect IQ field-visit media, Exotel recordings, WhatsApp media, and generated agreement/CAM assets. A new producer must register its owner map before it may write an object URL.
Ship a management command backfill_storage_locators --owner <owner-map-name> --batch-size 500 --dry-run and run it by owner map. It parses the old stored value through the shared normalizer, writes/upserts StorageLocator, then checks the object with HeadObject. The command reports separate counts and CSV rows for MAPPED, MISSING, INVALID, and AMBIGUOUS; it changes no original row and never copies or deletes an S3 object. Resolve every non-MAPPED PII record before enabling signed access for that owner map. Missing records stay inaccessible and return 404, rather than a guessed URL.
Before each owner-map backfill, deploy the writer change for that app: after a successful upload/copy, write the existing internal locator field and its StorageLocator in the same database transaction. Readers use the locator first; only while that map is in migration do internal readers normalize the old field on demand. Browser serializers return metadata and resource IDs throughout, so the backfill cannot reintroduce a public link.
Once an owner map has 100% MAPPED active records and its consumers use the locator, remove the app-local URL parser and public-URL constructor. Retain the old URL column as a private audit/source field until the app’s normal model migration window; do not expose it or accept it from clients. The registry remains the canonical bucket/key source, preventing a later CDN-domain rename from requiring database rewrites.
Do not cancel, restart, or rewrite Temporal histories. temporal_workflows has a 24-hour default execution timeout, but some document workflows can wait for later uploads; changing a workflow command sequence can make its stored history unreplayable. First deploy activity-level compatibility: every current activity accepts its existing reference_id, raw S3 URL, or new locator input and normalizes it before I/O. This preserves the existing workflow-to-activity command history while allowing the bucket to become private.
Before deployment, list and retain the workflow ID, run ID, workflow type, state, and document references for every running document-related execution on loan-application-queue; then monitor that same population until completion. The code already uses workflow.patched(...) for behavior changes. If a storage change requires adding/removing/reordering an activity call inside a workflow, guard the new branch with workflow.patched("storage-locator-v1"), keep the old branch for executions without that marker, and replay representative saved histories before rollout. Activity-only normalization does not need a workflow patch.
Treat in-flight temporary keys as valid until their owning workflow is terminal plus the existing retry window. Do not apply an S3 lifecycle deletion rule to temporary prefixes until the longest workflow/activity retry retention is confirmed from production configuration. The worker uses loan-application-queue; it must retain IAM read/copy permission to both temporary and permanent prefixes throughout the migration.
Replace the previous arbitrary HTTP fallback in two stages. Before strict mode, an allowlisted import activity accepts only named legacy vendor hosts, downloads with byte/time limits, writes a controlled temporary object, and passes its locator to the normal pipeline. Once the running-workflow audit shows no legacy external references, disable that importer and reject all non-S3 references. Unknown historic URLs fail the activity with a remediation code; they are never fetched from a worker.
Roll out each owner map through flags, in this order: LOCATOR_WRITE_ENABLED, LOCATOR_READ_ENABLED, BROWSER_URLS_REMOVED, then SIGNED_ACCESS_ENABLED. Enable one low-risk read-only map first, then the lead/BranchIQ map, then mutable BSA/AA and Temporal-dependent maps. Roll back only the active map by disabling its signing/serialization flag; do not make the S3 bucket public again.
An older workflow may have document_reference_id = "temp/APP123/pan/abc.jpg" in its saved state. After the compatibility deployment, its next OCR or copy activity converts that value to (primary-bucket, temp/APP123/pan/abc.jpg) and reads it with the worker’s IAM role. The workflow completes normally, creates the permanent APP123/pan_card/front.jpg object, and registers the new document row in StorageLocator. It never needs a browser signed URL. A BranchIQ employee later clicks View; the access endpoint finds the locator, checks the employee’s branch scope, and returns a five-minute CloudFront URL for that permanent key.
apis_djangoproject/document_upload/services.py — legacy primary S3 service and PdfMergeService.merge_pdf_urls; it currently returns public URLs and falls back to arbitrary HTTP downloads.apis_djangoproject/lead_collection/views/document_upload/client.py and apis_djangoproject/document_upload/views.py — upload-to-temp and Temporal copy/persist boundary; retain object keys across the workflow rather than browser URLs.apis_djangoproject/PraveshCAS/utils/s3_utils.py and apis_djangoproject/PraveshCAS/update_mapper.py — Nucleus attachment reads; replace URL parsing/HTTP fallback with trusted-locator IAM access.apis_djangoproject/branchiq/views.py, serializers.py, and urls.py — Pre-Login exposes raw document and bureau URLs and the two document/bureau detail views currently declare no visible permission class.Validate the two phases separately in a non-production AWS account with a known PII key. Goal 1: direct https://<bucket>.s3.<region>.amazonaws.com/<key> returns 403 while the deliberately unsigned CloudFront URL returns the expected file. Goal 2: the same unsigned CloudFront URL returns 403, a five-minute signed CloudFront URL renders the file and supports PDF range requests, and that URL returns 403 after expiry.
Run Django tests covering the shared locator parser, signer, each access endpoint’s allowed/foreign-application/foreign-branch/expired-signature cases, and the regression that URL-shaped external input is rejected by merge and Nucleus paths. Mock the CloudFront signer and S3 client; assert the observable HTTP status/body and the object key used, not implementation-specific helper calls.
For every owner map, run backfill_storage_locators --dry-run against a restored production-shaped snapshot, then run the real command twice. The first pass must produce only mapped/missing/invalid/ambiguous rows without modifying source URL columns; the second pass must be idempotent with zero new locators. Sample every mapped bucket/key with HeadObject, assert a missing locator produces 404 from its access endpoint, and block signed-access activation until PII INVALID and AMBIGUOUS counts are zero.
Replay saved histories for PAN, Aadhaar, generic document, grouped-image, BSA, AA, and Nucleus document workflows using the deployed compatibility code. Start one pre-cutover-style workflow with a temporary key and one with a legacy S3 URL, let both execute past copy/OCR, and confirm neither creates a non-determinism error nor needs a public S3 GET. List running executions before and after the deployment; their IDs must remain present until natural completion or cancellation by an existing business action.
Execute the Pravesh integration path in a staging fixture: multipart upload → temporary key → Temporal PAN/Aadhaar/bank-statement workflow → OCR/mask or password unlock → merge when applicable → PraveshCAS Nucleus payload. Assert the worker reads objects through S3 IAM and that no workflow payload or Django response contains a *.s3.*.amazonaws.com URL.
With a BranchIQ branch-manager account, open an application in the assigned branch, click View for an image and a PDF, and confirm each renders through the five-minute CloudFront capability. Attempt the same document ID with an application outside the account’s branch/state scope and expect 403; let a modal URL expire and confirm the single refresh renders it again. Inspect the pre-login list and document detail JSON to confirm it carries document IDs/metadata only.
For each bucket/access-class row in the final inventory, run its named UI/API consumer and worker job before moving its CloudFront behavior from public-CDN phase to signed-only phase. The release gate is zero direct S3 object GETs from browsers and zero raw S3 URLs in browser-facing API payloads, verified from CloudFront/S3 access logs and response-contract tests.
document_url, s3_document_url, file_url, and report URL database columns are retained as internal migration locators until each owning model is migrated to explicit bucket/key fields. If a stored locator cannot be normalized to an allowlisted bucket/key, deny browser access, report the record for remediation, and do not fall back to HTTP.