Downloading Tik Tok videos using Cloudflare Workers and Prompt API

Umut Gokbayrak
Umut Gokbayrak 16 October 2020
Author:

Tik Tok is irrepressible. It quickly gained popularity and derived a large number of startups around its ecosystem. From listing the popular videos to detection of influencers; many startups are trying to monetize this ecosystem.

You may also want to take advantage of the blessings of this world, but you have a big problem ahead of you.

As of October 2020, “Tik Tok does not provide an official API”.

Moreover, when you try to scrape Tik Tok's site, you'll soon notice how hard it is. First, in just a few minutes, your IP will be cut off from accessing the site. Even if you somehow manage to overcome this problem, you’ll realize that the page has been rendered on the client side and it is insufficient to scrape with the basics. Instead, you’ll need to launch a real browser, render and scrape with Selenium or any other similar framework.

Another solution would be to use the Prompt API's unofficial Tik Tok API, which provides unlimited access to Tik Tok's content, videos, hashtags and music.

Scraping Tik Tok Videos

Let's start by going to Tik Tok API's website and create a free Prompt API account.

Unofficial Tik Tok API

We get access to the API by pressing the “Start Using For Free” button in the upper right corner. Next, you may choose a plan that suits you (free tier available), fill in the sign up form (it just takes a few seconds) and start using the API right away.

Once again on the TikTok API web page, you can click the “Live Demo” button to review code samples and test the API with actual data.

Prompt API Live Demo Screen

Below you can find Tik Tok's Python code that captures 5 trending videos in the US. For other programming languages, samples are also available from API Documentation page or Live Demo.

import requests

url = "https://api.promptapi.com/tiktok/video/trending?region=us&lang=en&count=5"

payload = {}
headers= {
  "apikey": "YOUR API KEY"
}

response = requests.request("GET", url, headers=headers, data = payload)

status_code = response.status_code
result = response.text

The response when we execute this code (in its simplified form) is as follows.

[
  {
    ...
    "video": {
      "cover": "https://p16-sign-va.tiktokcdn.com/obj/tos-maliva-p-0068/0d488ae10c6b473a859c463e2ea2884d_1602398185?x-expires=1602788400&x-signature=W2yQKhPUcPzHEpHnn66bk%2BMcNT4%3D",
      "downloadAddr": "https://v16-web-newkey.tiktokcdn.com/44753539af834a252cc1557497feb035/5f88a731/video/tos/useast2a/tos-useast2a-ve-0068c001/4191252e8ae841ad815a151480dbfac4/?a=1988&br=3070&bt=1535&cr=0&cs=0&cv=1&dr=0&ds=3&er=&l=20201015134642010234088039421CB318&lr=tiktok_m&mime_type=video_mp4&qs=0&rc=anVtdDdmOnJkeDMzNjczM0ApPDloNzRlM2VkNzo7NGk1O2czYzU1NmVjYy5fLS0yMTZzczNfLjY1MjIxYWEzNC9hYWI6Yw%3D%3D&vl=&vr=",
      "duration": 14,
      "dynamicCover": "https://p16-sign-va.tiktokcdn.com/obj/tos-maliva-p-0068/c459c73cc3724846802f70512fc0876a_1602398184?x-expires=1602788400&x-signature=8XdXHcmocbDYKH27MZDZaMSyEFg%3D",
      "height": 1024,
      "id": "awesome",
      "originCover": "https://p16-sign-va.tiktokcdn.com/obj/tos-maliva-p-0068/ea1085e8131144aebb85e7e1fa21d063_1602398184?x-expires=1602788400&x-signature=NZ3wAqw%2BnOfFZeNbavTOS43tXiI%3D",
      "playAddr": "https://v16-web-newkey.tiktokcdn.com/44753539af834a252cc1557497feb035/5f88a731/video/tos/useast2a/tos-useast2a-ve-0068c001/4191252e8ae841ad815a151480dbfac4/?a=1988&br=3070&bt=1535&cr=0&cs=0&cv=1&dr=0&ds=3&er=&l=20201015134642010234088039421CB318&lr=tiktok_m&mime_type=video_mp4&qs=0&rc=anVtdDdmOnJkeDMzNjczM0ApPDloNzRlM2VkNzo7NGk1O2czYzU1NmVjYy5fLS0yMTZzczNfLjY1MjIxYWEzNC9hYWI6Yw%3D%3D&vl=&vr=",
      "ratio": "720p",
      "reflowCover": "https://p16-sign-va.tiktokcdn.com/obj/tos-maliva-p-0068/0d488ae10c6b473a859c463e2ea2884d_1602398185?x-expires=1602788400&x-signature=W2yQKhPUcPzHEpHnn66bk%2BMcNT4%3D",
      "shareCover": [
        "",
        "https://p16-sign-va.tiktokcdn.com/tos-maliva-p-0068/ea1085e8131144aebb85e7e1fa21d063_1602398184~tplv-tiktok-play.jpeg?x-expires=1602788400&x-signature=aJCXMU23%2F8R%2BYondtinvSg9E%2F%2FI%3D",
        "https://p16-sign-va.tiktokcdn.com/tos-maliva-p-0068/ea1085e8131144aebb85e7e1fa21d063_1602398184~tplv-tiktok-play2.jpeg?x-expires=1602788400&x-signature=bkkedBdmi8i3Y5nkq2DZP9ILFV8%3D"
      ],
      "width": 576
    },
    "vl1": false
  },
  ...
]

So far, everything is great. It was easy, wasn't it? However, when you try to copy & paste the playAddr (video) URL in your browser and proceed, unfortunately the videos will not play at all.

Tik Tok Video Error

Ouch! This is due to a security measure Tik Tok puts, in order to make it difficult to embed their videos anywhere. Tik Tok checks the Referer HTTP header for a valid Tik Tok URL, elsewise returns HTTP 403 status code that does not permit playing the content. Moreover, it's not possible to set the “Referer” header on the client side, as it will violate the HTTP protocol. The cleanest way is to write a backend that sets the HTTP Referer for you and acts as a proxy.

Re-streaming Tik Tok videos using Cloudflare Worker

Serverless Architecture and Cloud Functions have been on our agenda for some time. Amazon's Lambda Functions, Google's Google Functions, Azure Functions are the most popular players. But a lesser known but rather powerful candidate, Cloudflare's Workers is also perfect for the job. If you haven't checked it before, get started by creating a Cloudflare account.

After signing up and logging in, we proceed by selecting “Workers” under the “Products” section. We come across to a screen like this. (your workbench may be a little different)

Cloudflare Workers

By pressing the big blue “create a worker” button, you’ll head on to an IDE like screen where you will type code to get the job done.

Cloudflare Workers IDE

In terms of reference, our code is literally as follows:

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const { searchParams } = new URL(request.url)
  let url = searchParams.get('url')

  const referrer = "https://www.tiktok.com/foryou?lang=en"
  // Make the headers mutable by re-constructing the Request.
  request = new Request(request)
  request.headers.set("referer", referrer)

  // URL is set up to respond with dummy HTML
  let response = await fetch(url, request)

  // Make the headers mutable by re-constructing the Response.
  response = new Response(response.body, response)
  response.headers.set("referer", referrer)
  return response  
}

That code will get a video URL that you provide, from the query string, fetch it for you and re-stream it back. That's all. The Referer trick is available on line 12.


const referrer = "https://www.tiktok.com/foryou?lang=en"
request.headers.set("referer", referrer)

When we press the “Save and Deploy” button, Cloudflare will display the random URL where our “Worker” will be available. Eg. our unique worker URL is as follows:

https://small-violet-39bf.prop.workers.dev/

Now everything is ready. Now we can embed or run our video wherever we want.

The full URL to embed/play Tik Tok Videos, for our case, will be as follows:

https://small-violet-39bf.prop.workers.dev?url=[your_tik_tok_video_url]

Paste it in your browser and play it. Viola! Our video works flawlessly.

Playing Tik Tok Videos
Last words:

Highlights from this step by step tutorial are below:

  • Thanks to Prompt API, Tik Tok content can be accessed programmatically. There's no need to deal with rate limits or scraping.
  • We found out why the Tik Tok videos didn't work directly when pasted in the browser or embed in HTML5 video players.
  • We created a Cloudflare Worker and implemented a proxy that fetches and re-streams the Tik Tok videos.

Feel free to write any comments.

Share:
 
Umut Gokbayrak
Written by

Umut Gokbayrak

I turn ideas into software products