크롤링 방식에 대하여

[Laeng]

석진님께서 생각하시는 방향을 따라가고 지금까지 만들어주신 것을 최대한 활용을 하고자 했으나,
크롤링과 번역 만큼은 제 방식이 쉽게 가는 방식이라고 생각하여 제가 생각했던 게시글 크롤링 및 번역 방법에 관한 flow를 적습니다.

크롤링은 아래 3단계로 진행됩니다. 이와 관련하여 함께 의논하고 싶습니다.

Step 01

https://robertsspaceindustries.com/api/launcher/v3/content/news API 를 통해 게시글 조회

API 반환 값 예시

{
    "success": 1,
    "code": "OK",
    "msg": "OK",
    "data": {
        "rowcount": 8,
        "pagecount": 6,
        "startrow": 0,
        "resultset": [
            {
                "id": 19478,
                "publish_start": "2023-09-15 19:00:04",
                "time_created": "2023-09-15 18:32:53",
                "excerpt": "You asked. We're answering! Join us today for a live Q&A show with the Vehicle Gameplay team.",
                "title": "Star Citizen Live ",
                "url": "https://robertsspaceindustries.com/comm-link/transmission/19478-Star-Citizen-Live"
            },
            {
                "id": 19475,
                "publish_start": "2023-09-14 20:20:37",
                "time_created": "2023-09-14 15:41:49",
                "excerpt": "Join us today for our first part detailing the future of cargo, where we explore freight elevators, hangars, cargo boxes, and more!",
                "title": "Inside Star Citizen",
                "url": "https://robertsspaceindustries.com/comm-link/transmission/19475-Inside-Star-Citizen"
            },
            {
                "id": 19460,
                "publish_start": "2023-09-13 20:34:45",
                "time_created": "2023-09-06 18:07:52",
                "excerpt": "As we close in on Alpha 3.20’s Live release, many teams are finalizing their work on the patch’s various new features. Beyond the patch, significant progress is being made on content coming later in the year, so read on for the latest.",
                "title": "Star Citizen Monthly Report: August 2023",
                "url": "https://robertsspaceindustries.com/comm-link/transmission/19460-Star-Citizen-Monthly-Report-August-2023"
            },
            {
                "id": 19469,
                "publish_start": "2023-09-11 19:40:13",
                "time_created": "2023-09-11 11:07:13",
                "excerpt": "Happy Monday, everyone! Prepare to hoist the black flag as Pirate Week approaches, catch up on development news with the Monthly Reports, and raise a toast at Bar Citizens, This Week In Star Citizen!",
                "title": "This Week in Star Citizen ",
                "url": "https://robertsspaceindustries.com/comm-link/transmission/19469-This-Week-In-Star-Citizen"
            },
            {
                "id": 19468,
                "publish_start": "2023-09-08 19:48:38",
                "time_created": "2023-09-08 17:58:03",
                "excerpt": "Get ready to lock and load! Today, we're putting the Arena Commander Feature Team's gaming skills to the test as they play and showcase a variety of new game modes coming to Arena Commander in Alpha 3.20.",
                "title": "Star Citizen Live Gamedev",
                "url": "https://robertsspaceindustries.com/comm-link/transmission/19468-Star-Citizen-Live-Gamedev"
            },
            {
                "id": 19465,
                "publish_start": "2023-09-07 21:15:24",
                "time_created": "2023-09-07 16:56:42",
                "excerpt": "Are you feeling the need? The need for speed? Today we're taking you behind the scenes of our internal playtests featuring some of the new racing elements that will be available in Alpha 3.20, including an upcoming racing variant from Mirai.",
                "title": "Inside Star Citizen",
                "url": "https://robertsspaceindustries.com/comm-link/transmission/19465-Inside-Star-Citizen"
            },
            {
                "id": 19457,
                "publish_start": "2023-09-06 20:34:25",
                "time_created": "2023-09-05 21:44:35",
                "excerpt": "Every two weeks, we accompany the Roadmap update with a brief explanatory note to give you insight into the decision-making that led to any changes. This is part of an ...",
                "title": "Roadmap Roundup - September 6, 2023",
                "url": "https://robertsspaceindustries.com/comm-link/transmission/19457-Roadmap-Roundup-September-6-2023"
            },
            {
                "id": 19454,
                "publish_start": "2023-09-04 16:27:48",
                "time_created": "2023-09-04 08:31:45",
                "excerpt": "Happy Monday, everyone! We're down to our Final Four in Ship Showdown and taking a deeper look at Racing in Star Citizen in this week's ISC!",
                "title": "This Week in Star Citizen",
                "url": "https://robertsspaceindustries.com/comm-link/transmission/19454-This-Week-In-Star-Citizen"
            }
        ],
        "totalrows": 41,
        "pagesize": 8,
        "page": 1,
        "offset": 0,
        "estimatedrows": false
    }
}

Step 02

• resultset 데이터에서 각 게시글 요약 및 제목 수집, 링크 접속 후 게시글 크롤링

Step 03

• 크롤링된 게시글을 deepl을 통해 번역 후 게시