Getting more video ids than 30 fails about tiktok-api HOT 10 OPEN

@lhphat02

here is my code.

async def get_hashtag_videos(hash_tag, num_data=30):
    videos_data = []
    cursor = 0

    async with TikTokApi() as api:
        await api.create_sessions(ms_tokens=[ms_token], num_sessions=1, sleep_after=3, headless=False)
        tag = api.hashtag(name=hash_tag)

        while cursor <= num_data:
            async for video in tag.videos(count=30, cursor=cursor):
                print(video)
                video_data = video.as_dict
                videos_data.append(video_data) 
            cursor += 30

I actually use this solution for tag.videos, not for users.videos, and it works well. After checking the implementation for user.videos, it seems that this solution would work there too.

from tiktok-api.

@koonn Hi, thanks for your code, it works pretty well. But when I modify the code for crawling user.videos , it still not work, I think there're some problems with the TikTok's API or their policies about rate limits. Here's my code I modified from yours for user.videos :

async def get_user_videos(user_name, num_data=300):
    result = []
    cursor = 0

    async with TikTokApi() as api:
        await api.create_sessions(ms_tokens=[ms_token], num_sessions=1, sleep_after=3, headless=False)
        user = api.user(username=user_name)

        while cursor <= num_data:
            print(f"cursor: {cursor}")
            async for video in user.videos(count=30, cursor=cursor):
                print(video)
                video_data = video.as_dict
                result.append(video_data) 
            cursor += 30
            
    return result

Btw thanks for your solution. Have a nice day!

Which version of TikTokApi are you using?

Installing v6.2.2 broke user.videos() for me. Downgrading to 6.2.0 with pip install TikTokApi==6.2.0 --force-reinstall fixed the issue.

from tiktok-api.

When I set the count to 35 or higher, the same error occurs.

I've encountered a limitation where I cannot retrieve information for more than 35 videos at once. However, I found a workaround using the cursor argument in the users.videos endpoint.

This argument specifies the starting point for the count. By updating cursor and repeating the process, I can access information for more than 35 videos.

I hope this solution is helpful to you!

from tiktok-api.

@koonn
Hi, thanks for your code, it works pretty well. But when I modify the code for crawling user.videos , it still not work, I think there're some problems with the TikTok's API or their policies about rate limits. Here's my code I modified from yours for user.videos :

async def get_user_videos(user_name, num_data=300):
    result = []
    cursor = 0

    async with TikTokApi() as api:
        await api.create_sessions(ms_tokens=[ms_token], num_sessions=1, sleep_after=3, headless=False)
        user = api.user(username=user_name)

        while cursor <= num_data:
            print(f"cursor: {cursor}")
            async for video in user.videos(count=30, cursor=cursor):
                print(video)
                video_data = video.as_dict
                result.append(video_data) 
            cursor += 30
            
    return result

Btw thanks for your solution. Have a nice day!

from tiktok-api.

hey, try to look through this code ms_token = os.environ.get( "ms_token", None ) . Is your token actually located within your environment?
try to pass directly at first

from tiktok-api.

same here, more than 35 videos fail

from tiktok-api.

I've had mixed luck with trying to scrape large amounts at once. I'll post a modified version of your code adding the changes I've made that seem to work best, though I see a strong positive correlation between failure rate and post count (especially for profiles with >1000 posts).

Code

from TikTokApi import TikTokApi
import asyncio
import os

ms_token = os.environ.get(
    "ms_token", None
)  
async def user_example(username):
    async with TikTokApi() as api:
        await api.create_sessions(headless=False, ms_tokens=[ms_token], num_sessions=1, sleep_after=3)
        user = api.user(username)
        user_data = await user.info()
        post_count = user_data["userInfo"]["stats"].get("videoCount")

        async for video in user.videos(count=post_count):
            print(video)
            video = str(video) + "\n"
            with open('test1.json', 'a') as file:
                file.write(video)

if __name__ == "__main__":
    asyncio.run(user_example("truong_se"))

Breakdown

First, I added a username parameter to your user_example definition. This allows you to call this function for other usernames in a more streamlined fashion. It also cleans up your code elsewhere, as in user = api.user(username). (my own version of this function includes a manual limit: int = 0 in the definition for situations like where a user has a substantial number of non-public posts.)
Next, I added the post_count variable assignment. This will extract the number of videos from user_data in the previous line. Note: this number is not always accurate, as it reflects all posts - including private, which we can't scrape.
Finally, I added the username you had to the function call asyncio.run(user_example("truong_se")), reflecting the syntax update in the definition.

Hope this is helpful!

from tiktok-api.

I've had mixed luck with trying to scrape large amounts at once. I'll post a modified version of your code adding the changes I've made that seem to work best, though I see a strong positive correlation between failure rate and post count (especially for profiles with >1000 posts).

Code

from TikTokApi import TikTokApi
import asyncio
import os

ms_token = os.environ.get(
    "ms_token", None
)  
async def user_example(username):
    async with TikTokApi() as api:
        await api.create_sessions(headless=False, ms_tokens=[ms_token], num_sessions=1, sleep_after=3)
        user = api.user(username)
        user_data = await user.info()
        post_count = user_data["userInfo"]["stats"].get("videoCount")

        async for video in user.videos(count=post_count):
            print(video)
            video = str(video) + "\n"
            with open('test1.json', 'a') as file:
                file.write(video)

if __name__ == "__main__":
    asyncio.run(user_example("truong_se"))

Breakdown

First, I added a username parameter to your user_example definition. This allows you to call this function for other usernames in a more streamlined fashion. It also cleans up your code elsewhere, as in user = api.user(username). (my own version of this function includes a manual limit: int = 0 in the definition for situations like where a user has a substantial number of non-public posts.) Next, I added the post_count variable assignment. This will extract the number of videos from user_data in the previous line. Note: this number is not always accurate, as it reflects all posts - including private, which we can't scrape. Finally, I added the username you had to the function call asyncio.run(user_example("truong_se")), reflecting the syntax update in the definition.

Hope this is helpful!

My goal is to get all the user's video ids, I used my own method, surprisingly it works perfectly, it doesn't miss a single video.

This is my code:

driver = webdriver.Chrome() 
    user_link = "https://www.tiktok.com/@hieuthuhai2222"

    driver.get(user_link)

    time.sleep(10) 

    script = """let id_video = '';
    let lastScrollHeight = 0;
    let scrollCompleted = false;

    while (!scrollCompleted) {
        const currentScrollHeight = document.body.scrollHeight;

        if (currentScrollHeight === lastScrollHeight) {
            const divs = document.querySelectorAll('.css-1as5cen-DivWrapper');
            
            divs.forEach(div => {
                const link = div.querySelector('a');
                if (link) {
                    const href = link.getAttribute('href');
                    if (href.includes("/video/")) {
                        const id = href.split('/').pop();
                        id_video += id + ","; 
                    }
                }
            });

            scrollCompleted = true;
        } else {
            lastScrollHeight = currentScrollHeight;
            window.scrollTo(0, currentScrollHeight);
            await new Promise(resolve => setTimeout(resolve, 1000));
        }
    }

    return id_video;


    """

    ids = driver.execute_script(script)
    print(ids)

from tiktok-api.

Hi @koonn , I tried to use cursor but it didn't work. At the 35 first videos it worked but when the cursor start at 35 or higher, it don't return data anymore. Here's how I implemented:

async def get_user_videos(user_name):
  async with TikTokApi() as api:
    await api.create_sessions(ms_tokens=[ms_token], num_sessions=1, sleep_after=1, headless=False)
    user = api.user(user_name)
    count = 35
    cursor = 0

    while cursor < 350:
      print(f"Fetching videos from cursor {cursor}...")

      try:
        async for video in user.videos(count=count, cursor=cursor):
          print(video)
        cursor += count

      except Exception as e:
        print(f"Error: {e}")
        break

Have you tried using it? Then can I see your code please?
I'd be really appreciate that.

from tiktok-api.

@koonn Hi, thanks for your code, it works pretty well. But when I modify the code for crawling user.videos , it still not work, I think there're some problems with the TikTok's API or their policies about rate limits. Here's my code I modified from yours for user.videos :

async def get_user_videos(user_name, num_data=300):
    result = []
    cursor = 0

    async with TikTokApi() as api:
        await api.create_sessions(ms_tokens=[ms_token], num_sessions=1, sleep_after=3, headless=False)
        user = api.user(username=user_name)

        while cursor <= num_data:
            print(f"cursor: {cursor}")
            async for video in user.videos(count=30, cursor=cursor):
                print(video)
                video_data = video.as_dict
                result.append(video_data) 
            cursor += 30
            
    return result

Btw thanks for your solution. Have a nice day!

Which version of TikTokApi are you using?

Installing v6.2.2 broke user.videos() for me. Downgrading to 6.2.0 with pip install TikTokApi==6.2.0 --force-reinstall fixed the issue.

I personally downgraded and I could fetch more than 30+ videos per account without issues!

EDIT: wording

from tiktok-api.