← back to Tk 11331 Exec
scripts/__pycache__/scrape_momentum_feed.cpython-314.pyc
42 lines
+
���jB � �z � R t ^ RIt^ RIt^ RIt^ RItRt]P P RR4 t Rt
]P P ]P P ]P P ]4 4 4 t]P P! ]RR4 t. R&Ot. R'OtR(R ltR t. ]! 4 utt] EF� t^ t]R8 Ed� ]! ]]4 t]' g EM�] EF� t ] P R4 t ] P R 4 t ] P R 4 t ] e ] ! ] 4 ^ 8� d ] M] t!]PE ]! ] P R
4 4 ]! ] P R4 4 ]! ] P R 4 4 ]! ]!4 ]! ] P R
4 4 ]! ] P R4 4 ]! ] P R4 4 ]! ] P R4 4 ]! ] P R4 4 ]! ] P R4 4 ]! ]#! ] P R4 ]$4 ' d+ ] P R4 ;' g / P R4 M] P R4 4 . 4 EK� ]^�,
t]%! R] R] R]&! ]4 R2R R R 7 EK� ]%! R ] R 24 EK� ]'! ]R!4 ;_uu_ 4 t(](PS R"P! . R)O4 R#, 4 ] F* t*](PS R"P! ]*4 R#, 4 K, RRR4 ]%! R$]&! ]4 R%] 24 R# + ' g i L&; i)*u[
TK-11331 D3b — feed-first FULL Momentum scrape ($0, Meilisearch, no browser).
Pages the `redesign-colors` index sharded by category_id (each shard < 20k, so the
maxTotalHits=20000 offset cap never truncates), capturing every field the D3b recovery
join + manifest need. READ-ONLY against the feed; writes ONE local TSV artifact.
list_price = pre_discount_price if >0 else price (verified: on-sale rows carry the
discounted price in `price` and the true wholesale list in `pre_discount_price`;
non-sale rows carry list in `price` with pre_discount_price=0).
OUTPUT: data/momentum_feed_full.tsv (tab-separated, one row per colorway)
cols: alt_product_description, number, preferred_color_number, list_price,
is_roll_price, uom, base_width, category_name, pattern_name,
preferred_color_name, product_line_code
Nz/https://ms-e886719d86e7-4256.sfo.meilisearch.io�MOMENTUM_MS_KEY�@95fe8376edc78d49e787db49edda68426b21649f6271f49e2f1412118612fbd6zredesign-colors�datazmomentum_feed_full.tsv�number�preferred_color_number�alt_product_description�price�pre_discount_price�
is_roll_price�UOM�
base_width�
category_name� pattern_name�preferred_color_name� product_linec �� � \ P ! R RRRV 2RVRVR\ /4 P 4 p\ P
P
\ R\ R2VR R
\ 2R R /R
R7 p\ P
P V^<R7 ;_uu_ 4 p\ P ! VP 4 4 P R. 4 uuRRR4 # + ' g i R# ; i)�q� �filterzcategory_id = �limit�offset�attributesToRetrievez /indexes/z/search�
AuthorizationzBearer z Content-Typezapplication/json�POST)r �headers�method)�timeout�hitsN)�json�dumps�FIELDS�encode�urllib�request�Request�HOST�INDEX�KEY�urlopen�loads�read�get)�cidr r �body�req�rs &&& �H/Users/macstudio3/Projects/tk-11331-exec/scripts/scrape_momentum_feed.py�pager1 # s� � � �:�:�s�B��N�3�%�*@�'�5� ��)?��I� J�JP�&�(� �
�.�.�
�
�D�6��5�'��!A��$���u�o�~�GY�Z�� !� �C�
� � � � ��R� � 0� 0�A��z�z�!�&�&�(�#�'�'���3�
1� 0� 0� 0�s �4C� C+ c � � V f RM
\ V 4 P RR4 P RR4 P 4 # )Nr � � �
)�str�replace�strip)�ss &r0 �cleanr: , s5 � ��)�B��Q�� 0� 0��s� ;� C� C�D�#� N� T� T� V�V� i N �codez
cat z offset z total-rows z r T)�end�flushz [cat z done]�wr3 r5 z
WROTE z feed rows -> )� � � � � � � � ) r r r r r r
r r r
r r r )�� ) r r r �
list_pricer
�uomr r
r r �product_line_code)+�__doc__r �os�time�urllib.requestr" r% �environr+ r' r& �path�dirname�abspath�__file__�HERE�join�OUT�CAT_IDSr r1 r: �set�rows�seen_idsr, �offr �h�hidr �pdp�float�lp�append�
isinstance�dict�print�len�open�f�writer/ � r; r0 �<module>rk s� ��� &� %�8��
�z�z�~�~�'�J�L���� �w�w���r�w�w���r�w�w���x�'@�A�B�� �w�w�|�|�D�&�":�;��
"��
S��4�W� �S�U���h� �C�
�C�
��+��C��~��� ��A��%�%��/�C��E�E�'�N�E��%�%�,�-�C� � �U�3�Z�!�^��%�B� �K�K��a�e�e� 5�6�7��a�e�e�H�o�&��a�e�e� 4�5�6��b� ��a�e�e�O�,�-��a�e�e�E�l�#��a�e�e�L�)�*��a�e�e�O�,�-��a�e�e�N�+�,��a�e�e� 2�3�4��:�a�e�e�Tb�Nc�ei�Cj�Cj�q�u�u�^�,�2�2��7�7��?�pq�pu�pu� wE� qF� G� �
� �$ �s�
��
����X�c�U�,�s�4�y�k��E�2�UY�Z�Z� �G�C�5��
� �5 �8
�#�s�^�^�q��G�G�D�I�I� C�
D�FJ�
K� L� �� ���� � �!� �t�#�$� � � ���T�� �>�#��/� 0�
�^�s
�2AL*�* L: