← back to Commercialrealestate

scripts/__pycache__/fetch-dre-mlo.cpython-314.pyc

113 lines

+
i�j$����Rt^RIt^RIt^RIt^RIt^RIt]PP]PP]PP]
444t]PP]RR4t
RtRtRt0RkRkR	kR
kRkRkR
kRkRkRkRkRkRkRkRkRkRkRkRkRkRkRkRkRkRkR kR!kR"kR#kR$kR%kR&kR'kR(kR)kR*kR+kR,kR-kR.kR/kR0kR1kR2kR3kR4kR5kR6kR7kR8kR9kR:kR;kR<kR=kR>kR?kR@kRAkRBkRCkRDkREkRFkRGkRHkRIkRJkRKkRLkRMkRNkROkRPkRQkRRkRSkRTkRUkRVkRWkRXkRYkRZkR[kR\kR]kR^kR_kR`kRakRbkRckRdkRekRfkRgkRhkRikRjkRkkRlkRmkRnkRokRpkRqkRrkRskRtkRukRvkRwkRxkRykRzkR{kR|kR}kR~kRkR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�kR�ktR�R�R�R�R�R�/tR�tR�tR�tR�t]R�8Xd
]!4R#R#)�u�
fetch-dre-mlo.py — Ingest CA DRE's daily "MLO List" (individual real-estate licensees
with a SAFE Act / NMLS mortgage-loan-originator endorsement) into data/loan-officers.json.

PERMITTED SOURCE (DATA_POLICY.md rule 1 = official_bulk_download):
  Page:  https://www.dre.ca.gov/Licensees/ExamineeLicenseeListDataFiles.html
  File:  https://secure.dre.ca.gov/datafile/mlo_list.xls   (updated daily by CA DRE)

SCOPE (Steve, 2026-08-20): ACTIVE, INDIVIDUAL (Broker/Salesperson), LA-County only.
COMPLIANCE:
  - Business/license fields only. address_line_1/2 + zip are DROPPED (rule 3 — no home address).
  - phone_number is kept: it is published by a government source in this public file (rule 3 carve-out).
  - Every row carries a real NMLS # + CalBRE # + the DRE source_url it came from (rule 4).
  - Merges with (never clobbers) the verified firm-roster officers already in the file; dedup by NMLS #.
Run:  python3 scripts/fetch-dre-mlo.py [--download]   (default reuses /tmp/mlo_list.xls if present)
N�datazloan-officers.jsonz/https://secure.dre.ca.gov/datafile/mlo_list.xlszChttps://www.dre.ca.gov/Licensees/ExamineeLicenseeListDataFiles.htmlz/tmp/mlo_list.xlszAGOURA HILLS�ALHAMBRA�ARCADIA�ARTESIA�AVALON�AZUSAzBALDWIN PARK�BELLzBELL GARDENS�
BELLFLOWERz
BEVERLY HILLS�BRADBURY�BURBANK�	CALABASAS�CARSON�CERRITOS�	CLAREMONT�COMMERCE�COMPTON�COVINA�CUDAHYzCULVER CITYzDIAMOND BAR�DOWNEY�DUARTEzEL MONTEz
EL SEGUNDO�GARDENA�GLENDALE�GLENDORAzHAWAIIAN GARDENS�	HAWTHORNEz
HERMOSA BEACHzHIDDEN HILLSzHUNTINGTON PARK�INDUSTRY�	INGLEWOOD�	IRWINDALEzLA CANADA FLINTRIDGEzLA HABRA HEIGHTSz	LA MIRADAz	LA PUENTEzLA VERNE�LAKEWOOD�	LANCASTER�LAWNDALE�LOMITAz
LONG BEACHzLOS ANGELES�LYNWOOD�MALIBUzMANHATTAN BEACH�MAYWOOD�MONROVIA�
MONTEBELLOz
MONTEREY PARK�NORWALK�PALMDALEzPALOS VERDES ESTATES�	PARAMOUNT�PASADENAzPICO RIVERA�POMONAzRANCHO PALOS VERDESz
REDONDO BEACHz
ROLLING HILLSzROLLING HILLS ESTATES�ROSEMEADz	SAN DIMASzSAN FERNANDOzSAN GABRIELz
SAN MARINOz
SANTA CLARITAzSANTA FE SPRINGSzSANTA MONICAzSIERRA MADREzSIGNAL HILLzSOUTH EL MONTEz
SOUTH GATEzSOUTH PASADENAzTEMPLE CITY�TORRANCE�VERNON�WALNUTzWEST COVINAzWEST HOLLYWOODzWESTLAKE VILLAGE�WHITTIERzNORTH HOLLYWOODzVAN NUYSzSHERMAN OAKSzSTUDIO CITY�ENCINO�TARZANAzWOODLAND HILLSzCANOGA PARK�RESEDA�
NORTHRIDGEz
GRANADA HILLS�
CHATSWORTHz
SUN VALLEYz
PANORAMA CITY�SYLMAR�PACOIMAz
MISSION HILLSzVALLEY VILLAGE�VALENCIAzSTEVENSON RANCH�NEWHALL�SAUGUSzCANYON COUNTRY�CASTAICzMARINA DEL REYzPLAYA VISTAz
PLAYA DEL REY�VENICE�WESTCHESTERzPACIFIC PALISADES�	BRENTWOOD�	HOLLYWOODz
WEST HILLS�WINNETKAzLAKE BALBOA�ARLETAzLA CRESCENTA�MONTROSE�TUJUNGA�ALTADENA�TOPANGA�ACTONz
AGUA DULCEzQUARTZ HILL�
LITTLEROCKzLAKE HUGHESzLEONA VALLEYz	VIEW PARKz
WINDSOR HILLSzLADERA HEIGHTSzHACIENDA HEIGHTSzROWLAND HEIGHTS�VALINDAzWEST PUENTE VALLEY�CITRUSzCHARTER OAKzEAST LOS ANGELESz
WEST WHITTIERzSOUTH WHITTIER�WILLOWBROOK�FLORENCEzWALNUT PARK�LENNOXzDEL AIRE�WESTMONTzLAKEVIEW TERRACEzSHADOW HILLS�SUNLANDzPORTER RANCHzHARBOR CITY�
WILMINGTONz	SAN PEDRO�Brokerz$Real Estate Broker (MLO endorsement)�Salespersonz)Real Estate Salesperson (MLO endorsement)�Officerz%Real Estate Officer (MLO endorsement)c��\V\4'd!VPRR4P4#VfR#\V4#)��)�
isinstance�str�replace�strip��ss&�scripts/fetch-dre-mlo.py�cleanr\%s=��,6�q�#�,>�,>�1�9�9�V�R� �&�&�(�a�1�9�R�a�Z]�^_�Z`�a�c��RP\V4P44pV'd'VP4'dVP	4#T#)� )�joinr\�split�isupper�titlerYs&r[�	titlecaserd(s8������q����!�"�A��a�i�i�k�k�1�7�7�9�0�q�0r]c�Vaa"a#�R\P9g*\PP	\
4'g�\
R\R24\PP\RR/R7p\PPV^ZR7;_uu_4o#\\
R4;_uu_4pVPS#P44RRR4RRR4^R	IHpVP#\
4P%^4P'4pV^,Uu.uFp\)V4NK	pp\+V4UUu/uFwrgWvbK		uppo"\,P,P/\,P0P24P5R
4p.p	\74p
VR,EF�o#V"V#3RlpV!R
4V!R4V!R4r�pV!R4V!R4P94ppV\:9dKMV
R8XdVR8XgK]V'dV\<9dKrW�9dKzV
P?V4\AV!R44\AV!R44\)V!R44pppRPCRVVV344PE4;'gTpRS"9dS#S"R,,MRp\GVR4'dVPI4M
\)V4pV!R4pV	PK/RRV,bRVbR \AV!R!44bRVbR"\LPOW�4bR#V!R$4bR%RbR&\AV4bR'R(bRVbR
VbR)V
bR*VbR+R,bR-\PbR.R/VR0V2bR1Vb4EK�	\RPT!\\V44pVPOR2.4pVUu0uFpVPOR4kK	ppV	Uu.uFpVR,V9gKVNK	ppVV,pVPYR3R47VR5,pR6VR+&R7VR8&\[V4VR9&VVR1&R:VR;&R<VR=&VPOR>.4Uu.uFpVPOR84R?8XgKVNK	p pR+R@RA\PRB\R8RCR;RDR9\]REV44/.V ,VR>&VUau.uFNo\^;QJdV3RFlRT4F'gKRGM	RHM!V3RFlRT44'gKESR,NKP	p!pV!'dM\P`PRI\[V!4RJV!RK,RL24\Pb!^4\RPd!R5VR2V/\\VRM4^RN7\
RO\[V	4RP\[V424\
RQ\[V4RR\[V424\
RS4R# +'giEL;i +'giEL;iuupiuuppiuupiuupiuupiuupi)Uz
--downloadzdownloading z ...z
User-AgentzMozilla/5.0)�headers)�timeout�wbN)�CalamineWorkbookz%Y-%m-%dT%H:%M:%SZ:�NNc�v<�VS9d1SV,\S48d\SSV,,4#R#)rT)�lenr\)�k�ix�rs&��r[�<lambda>�main.<locals>.<lambda>;s-����b��R��U�S��V�^�e�A�b��e�H�o�K��Kr]�calbre_license_type�calbre_license_status�nmls_status�nmls_id�	city_name�Licensed�APPROVED�firstname_secondary_name�lastname_primary_name�name_suffixr_c3�8"�TFq'gKVx�K	R#5i�N�)�.0�ps& r[�	<genexpr>�main.<locals>.<genexpr>Ds���;�#5�a����#5�s�	�
�calbre_expiration_daterT�	isoformat�calbre_license_id�idznmls-�name�company�sponsor_namerc�business_phone�phone_number�business_email�city�state�CA�license_status�license_expires�sourceuCA DRE — MLO List (daily)�
source_url�
identifierzNMLS #u · CalBRE #�retrieved_at�officersc�n�VPR4;'gRVPR4;'gR3#)r��~r�rT)�get)�os&r[rprqbs)�����v��!5�!5�#�q�u�u�V�}�7J�7J�� Kr])�key�metazaCA DRE MLO List (official bulk download) + firm-published public rosters (first-party public web)z/official_bulk_download + first_party_public_web�
access_method�countuiCalifornia — Los Angeles County; active, individual MLO-endorsed licensees (Broker/Salesperson/Officer)�scopeu�Individual mortgage loan officers are added ONLY from officially-permitted sources: the CA DRE daily MLO List (official_bulk_download) and firms' OWN public loan-officer rosters (first_party_public_web). NMLS Consumer Access is NEVER scraped. Every record carries a real public NMLS license number + source_url; business/license fields only — no home/mailing addresses, no personal emails. Phone is the government-published contact from the DRE public file. See docs/DATA_POLICY.md.�policy�sources�first_party_public_webu#CA DRE — MLO List (updated daily)�page_url�file_url�official_bulk_downloadz:LA County, active, individual (Broker/Salesperson/Officer)c3�"�TF9p\VPRR44PR4'gK5^x�K;	R#5i)r�rTzCA DREN)rVr��
startswith)rr�s& r[r�r�vs0���Z��1�C����h��0C�,D�,O�,O�PX�,Y�Q�Q��s
�2A�
Ac3�,<"�TF	qS9x�K	R#5ir}r~)rrmr�s& �r[r�r�zs����3|�D{�q��F�D{�s�TFu$ABORT — PII leak (address/zip) in z records, e.g. :N�Nz; nothing written.
�w)�indentz*DRE LA-County active individuals matched: z | new added (dedup by NMLS): z firm-roster officers preserved: z | TOTAL officers: u1PII guard (address/zip fields present): clean ✓)�address_line_1�address_line_2�zip_code�zip)3�sys�argv�os�path�exists�	XLS_CACHE�print�XLS_URL�urllib�request�Request�urlopen�open�write�read�python_calamineri�	from_path�get_sheet_by_index�	to_pythonr\�	enumerate�datetime�now�timezone�utc�strftime�set�upper�INDIVIDUAL_TYPES�	LA_COUNTY�addrdr`rX�hasattrr��append�TITLEr��SRC_PAGE�json�load�DATA�sortrl�sum�any�stderr�exit�dump)$�req�fri�rows�x�hdr�i�hr��dre�seen�g�lt�st�ns�nmlsr��first�last�sufr��exp�calbre�cur�existingr��
existing_nmls�d�addedr�r�rZ�firm_sources�leakedrnros$                         `        @@r[�mainr�,s�����s�x�x��r�w�w�~�~�i�'@�'@�
��W�I�T�*�+��n�n�$�$�W�|�]�6S�$�T��
�^�^�
#�
#�C��
#�
4�
4��4�	�4�;P�;P�TU�
�G�G�A�F�F�H��<Q�
4�0��%�%�i�0�C�C�A�F�P�P�R�D�!�!�W�
%�W��5��8�W�C�
%�$�S�>�	*�>�4�1�!�$�>�	*�B�
�
�
�
�
�� 1� 1� 5� 5�
6�
?�
?�@T�
U�C�
�C��5�D�
�"�X�X��K���,�-�q�1H�/I�1�]�K[����y�\�1�[�>�#7�#7�#9�d��
�%�%�x��j� �R�:�%5���t�9�,�h��<�������$�Q�'A�%B�C�Y�q�Qh�Oi�Ej�lq�rs�uB�sC�mD�S�t���x�x�;�E�4��#5�;�;�A�A�C�K�K�t��1I�R�1O�a��+�,�-�UW��!(��k�!:�!:�c�m�m�o��c�
���&�'���
�
�
��'�D�.�
��D�
�
�y��>�!2�3�
�
�t�	
�

�U�Y�Y�r�&�
�
�a��/�

�
�b�
�
�I�d�O�
�
�T�
�
 ��
�
"�2�
�
�b�
�
�s�
�
�3�
�
�(�
� 
�F�4�&��V�H�=�!
�"
�C�#
�	��F�)�)�D��J�
�C��w�w�z�2�&�H�/7�8�x�!�Q�U�U�9�%�x�M�8��A��1�q��|�=�@�Q�Q��E�A��%��H��M�M�K�M�L��v�;�D�x�D��N�M�D�����M�D��M��D����D��M�o�D��N�
 $�x�x�	�2�6�m�6�!�!�%�%��:P�Tl�:l�A�A�6�L�m��7��H��G��1��M���Z��Z�Z�
��
��D��O�%-�
}�H�q���3|�D{�3|����3|�D{�3|�0|�l�a�	�l�l�H�F�
}�
��
�
���?��F��}�O�\b�ce�\f�[g�g{�|�}�������I�I�v�t�Z��2�D��s�O�A�N�	�6�s�3�x�j�@^�_b�ch�_i�^j�
k�l�	�,�S��]�O�;N�s�S[�}�o�
^�_�	�
=�>�c<Q�;P�;P��
4�
4�
4��&��	*��T9��A��n��~sf�%W8�> W$	�W8�1X�
X�X�$X�9X�!X!�>X!�<X&�X&�#X&�
X&�$W5
�/	W8�8X		�__main__)rOrPrQ)�__doc__r�r�r�r��urllib.requestr�r��dirname�abspath�__file__�ROOTr`r�r�r�r�r�r�r�r\rdr��__name__r~r]r[�<module>r�s���� ��	�
��	�w�w���r�w�w���r�w�w���x�'@�A�B��	�w�w�|�|�D�&�"6�7��
;��P���	�
��
��
�#�
�$-�
�.6�
�7>�
�?M�
�NT�
�Uc�
�dp�
�rA�
�BL�
�MV�
�Wb�
�ck�
�lv�
�wB�
�CM�
�NW�
�X`�
�ai�
�jw�
�xE�
�FN�
�OW�
�Xb�
�co�
�py�
�zD�
�EO�
�Pb�
�cn�
�o~�
�M�
�N_�
�`j�
�kv�
�wB�
�CY�
�Zl�
�mx�
�yD�
�EO�
�PZ�
�[f�
�gq�
�rz�
�{G	�
�H	U	�
�V	_	�
�`	h	�
�i	z	�
�{	D
�
�E
O
�
�P
\
�
�]
l
�
�m
v
�
�w
A�
�BX�
�Yd�
�eo�
�p}�
�~F�
�G\�
�]l�
�m|�
�}T
�
�U
_
�
�`
k
�
�l
z
�
�{
H�
�IU�
�Ve�
�fx�
�yG�
�HV�
�Wd�
�eu�
�vB�
�CS�
�Ta�
�bl�
�mu�
�v~�
�L�
�M]�
�^p�
�q{�
��
��
�+�
�,9�
�:B�
�CL�
�M]�
�^k�
�lt�
�vB�
�CR�
�S_�
�`l�
�m|�
�}E�
�FO�
�P_�
�`p�
�q{�
�|M�
�NW�
�X`�
�aq�
�r{�
�|L�
�MZ�
�[j�
�ks�
�tA�
�BU�
�Va�
�bm�
�nz�
�{E�
�FS�
�T\�
�]k�
�lv�
�w@�
�AK�
�LU�
�V]�
�^j�
�kx�
�yE	�
�F	S	�
�T	b	�
�c	n	�
�o	~	�
�	O
�
�P
b
�
�c
t
�
�u
~
�
�
S�
�T\�
�]j�
�k}�
�~M�
�N^�
�_l�
�mw�
�xE
�
�F
N
�
�O
Y
�
�Z
d
�
�e
w
�
�x
F�
�GP�
�Q_�
�`m�
�nz�
�{F�
�	�
�9�=�Ju�xA�Cj�	k��7��b�1�U?�n�z���F�r]