← back to Stroheim Onboard
__pycache__/scrape_details.cpython-314.pyc
34 lines
+
{�`j� � � � R t ^ RIt^ RIt^ RIt^ RIt^ RIt^ RIHtHt ^ RI t
Rt R R lt R t
R tR t]R8X d
]! 4 R# R# )
u
Stroheim detail-spec enrichment — feed-first, $0 local.
Reads data/listing.jsonl, fetches each public detail page, extracts
width / repeat / content(material) / book(collection) / type.
Writes data/details.jsonl (append-safe: skips skus already done).
N)�ThreadPoolExecutor� as_completedzsMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17 Safari/605.1.15c �� � \ V4 F p \ P P V R \ /R7 p\ P P V^ R7 ;_uu_ 4 pVP
4 P RR4 uuRRR4 u # R# + ' g i K� ; i \ d: Y!^,
8X d R# \ P ! RT^, , 4 K� i ; i)z
User-Agent)�headers)�timeoutzutf-8�replaceN� g333333�?) �range�urllib�request�Request�UA�urlopen�read�decode� Exception�time�sleep)�url�tries�t�req�rs && �scrape_details.py�fetchr s� � �
�5�\�� &��.�.�(�(��|�R�6H�(�I�C�� � �'�'��R�'�8�8�A��v�v�x���w� �:� 9�8� � �
9�8�8��� &��A�I�~�� �J�J�s�a�!�e�}� %� &�s6 �AB%� B�
B%� B" � B%�"B%�%C)�$C)�(C)c �0 � V F� p\ P ! V R2V \ P 4 pV' g K6 \ P ! RRVP ^4 4 P R4 pV' g Kv VP
4 R9 g K� Vu # R# )zCFind a spec value following any of the label words in a tech block.z*\s*[:<][^>]*>?\s*</?[^>]*>?\s*([^<]{1,60})�\s+� u : N)r zn/a)�re�search�I�sub�group�strip�lower)�html�labels�lab�m�vs &* r �specr* ss � ��� �I�I�#�� J�K�T�SU�SW�SW� X�� �1����v�s�A�G�G�A�J�/�5�5�f�=�A��q�Q�W�W�Y�k�1��� � � c �� � R pV P R4 pVR8w d WVR, pT;' g T p/ p\ VR4 VR& \ VRR4 VR& \ VR4 VR & \ VR
R R 4 VR
& \ VR4 VR& \ VRR4 VR& \ P ! RV 4 pV' d6 \ P ! RRVP ^4 4 P
4 MRVR& V# )r � product_techi� �Width�widthzVertical Repeat�Repeat�repeat_vzHorizontal Repeat�repeat_h�Content� Composition�Material�material�Match�
match_type�Book�
Collection�bookz&<div class="type">\s*([^<]+?)\s*</div>r r N�
type_label�����)�findr* r r r! r" r# )r% �tech�i�region�d�tms & r � parse_detailrD % s� � �
�D� � � �.�!�A��B�w��a��f�~��
�\�\�T�F�
�A��f�g�&�A�g�J���!2�H�=�A�j�M���!4�5�A�j�M����M�:�F�A�j�M��6�7�+�A�l�O��V�V�\�2�A�f�I� ���<�d� C�B�BD�b�f�f�V�S�"�(�(�1�+�6�<�<�>�$�A�l�O� �Hr+ c
�B � \ R 4 U u. uF p \ P ! V 4 NK pp \ 4 p\ P
P
R4 ' d@ \ R4 F0 p VP \ P ! V 4 R, 4 K2 V Uu. uF q3R, V9 g K VNK pp\ P P R\ V4 R\ V4 R24 ^ p\ RR4 ;_uu_ 4 p\ ^
R7 ;_uu_ 4 pV Uu/ uF q7P \ VR, 4 VbK" pp\# V4 F� p W�, pV P% 4 p
RVR, /p V
' d V P' \) V
4 4 VP \ P* ! V 4 R , 4 VP- 4 V^,
pV^�, ^ 8X g K� \ P P R
V R \ V4 R 24 K� R R R 4 R R R 4 \/ R
V R24 R # u up i \ P \ 3 d EK i ; iu upi u upi + ' g i L[; i + ' g i Lf; i)zdata/listing.jsonlzdata/details.jsonl�mfr_skuz details: z to fetch (z already done)
�a)� max_workers� product_url�
z �/NzDONE details: z fetched)�open�json�loads�set�os�path�exists�add�JSONDecodeError�KeyError�sys�stderr�write�lenr �submitr r �result�updaterD �dumps�flush�print) �l�listing�doner �todo�n�out�ex�futs�fr% �recs r �mainrj 8 s� � �&*�+?�&@�A�&@��t�z�z�!�}�&@�G�A� �5�D� �w�w�~�~�*�+�+��*�+�A�
�������A� �y�1�2� ,�
� ;�w�!�I�,�d�":�A�A�w�D� ;��J�J���y��T�� �;�s�4�y�k�AQ�R�S� �A�
�"�C� (� (�C�
�B�
/�
/�2�CG�H�4�a�I�I�e�Q�}�%5�6��9�4�D�H�!�$�'���G���x�x�z�� �!�I�,�/����J�J�|�D� 1�2�� � �$�*�*�S�/�D�0�1�� � � ��Q����s�7�a�<��J�J�$�$�r�!��A�c�$�i�[��%;�<� (� 0�
)�
�N�1�#�X�
&�'��3 B�� �(�(�(�3�
��
�� ;��
I� 0�
/��
)� (�s` � I �-,I�!I1�5I1� J� I; �%&I6 � B%I; �53I; �(J�I.�-I.�6I; �; J
�J� J �__main__)� )�__doc__r rM rV rP r �concurrent.futuresr r �urllib.requestr
r
r r* rD rj �__name__� r+ r �<module>rr sJ ��� � � � ?� �7��
��
�&(�8 �z���F� r+