In this blog I will be creating a simple node CLI tool for downloading Youtube videos with an option to automatically convert them to a specified audio or video format. Among others, this guide will cover the creation of global node modules and the usage of Typescript.
The code for this project can be found at https://github.com/learningdollars/martas-youtubedownloader.
Prerequisites
To start the project, you need to have Node.js and npm installed. All the available install options are listed here.
Part 1 – Project Setup
I will initialize the project by running npm init after which a series of inputs will be requested. Here, I will name my project “yt-converter”, add a short description and keywords. To skip this part and use the default values, run npm init with the -y option. After finishing, package.json file will be generated that will look like this:
{
"name": "yt-converter",
"version": "1.0.0",
"description": "A simple tool for downloading youtube videos and converting them to a specified format.",
"main": "index.js",
"scripts": {
"test": "echo \"No tests specified.\" && exit 0",
},
"repository": {
"type": "git",
"url": "git+https://github.com/learningdollars/martas-project.git"
},
"keywords": [
"youtube",
"ffmpeg",
"typescript",
"node"
],
"author": "martas",
"license": "ISC",
"bugs": {
"url": "https://github.com/learningdollars/martas-project/issues"
},
"homepage": "https://github.com/learningdollars/martas-project#readme",
"dependencies": {
}
}
The next step is the installation of development dependencies:
npm install ts-node-dev typescript --save-dev
I’ll update package.json scripts field with the following lines:
"scripts": {
"test": "echo \"No tests specified.\" && exit 0",
"start": "node ./dist",
"build": "npx tsc",
"watch": "tsnd --respawn ./src"
},
The build command runs the typescript compiler and the watch command runs the development server. To be able to run typescript compiler we will need tsconfig.json file, created by running: tsc --init or npx tsc --init. I will replace the default file content with the following:
{
"exclude": [
"tests",
"node_modules",
"dist"
],
"compilerOptions": {
/* Basic Options */
"target": "es5",
"resolveJsonModule":true,
/* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017','ES2018' or 'ESNEXT'. */
"module": "commonjs",
/* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', or 'ESNext'. */
"lib": [
"es2015"
],
/* Specify library files to be included in the compilation. */
"declaration": true, /* Generates corresponding '.d.ts' file. */
"outDir": "./dist",
/* Strict Type-Checking Options */
"strict": true,
/* Enable all strict type-checking options. */
"noImplicitAny": true,
/* Additional Checks */
"noUnusedLocals": true,
/* Report errors on unused locals. */
"noUnusedParameters": true,
"esModuleInterop": true /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
},
}
In the last step of this section, I will create ./src/index.ts file as an entry point and add a .gitignore file:
/node_modules /dist
Part 2 – Project Walkthrough
I will start the development by creating YtConverter class and adding the following code to index.ts file:
import YtConverter from "./YtConverter"
export function init(urls?: Array<string>, dir?: string, convert?: boolean, format?: string) {
new YtConverter().init(urls, dir, convert, format).catch((err: Error) => {
console.error("ERROR", err)
process.exit(1)
})
}
export default class YtConverter {
/**
* @param {Array<string>} urls
* - array of video urls to download
* @param {string} dir - optional
* - destination directory - default /home/{user}/videos
* @param {boolean} convert - optional
* - should the videos be converted - default: false
* @param {string} format - optional
* - format to convert into - default: "mp3"
* Initializes video download / conversion
*/
async init(urls?: Array<string>, dir?: string, convert?: boolean, format?: string, update?: boolean) {
}
}
YtConverter.init function will take 5 (optional) paramaters:
urls– and array of Youtube video URLsdir– download destination directoryconvert– boolean value determining weather the video should be convertedformat– format to convert intoupdate– should ffmpeg be checked for updates (ffmpeg binary will be downloaded later in this guide)
You can start the development server by running npm run watch. For now, multiple errors will be displayed in the console, as all the parameters of the YtConverter.init function are not used and the "noUnusedParameters" rule is set to true in tsconfig.json file. Also, Node.js process and console objects will produce type errors, that can be solved by installing type definitions for Node.js: npm install @types/node --save-dev.
Next, I’ll add a new class Args in Args.ts file that will parse command-line arguments and set them as instance fields. Here I will need yargs module and its types, so I will run: npm install yargs @types/yargs .
import yargs from "yargs"
export default class Args {
private _urls!: Array<string | number>
private _dir!: string
private _convert!: boolean
private _format!: string
private _update!: boolean
constructor() {
this.parseCliArgs()
}
parseCliArgs() {
const args = yargs
.option('url', {
alias: "u",
describe: "video url to download",
array: true,
})
.option('dir', {
alias: "d",
default: `/home/${require("os").userInfo().username}/videos`,
describe: "destination directory",
string: true
})
.option('convert', {
alias: "c",
default: false,
describe: "convert to mp3",
boolean: true
})
.option('format', {
alias: "f",
default: "mp3",
describe: "format to convert into",
string: true
})
.option('update-ffmpeg', {
alias: "uf",
default: false,
describe: "update ffmpeg",
boolean: true
})
.alias("h", "help")
.help('help')
.argv
if (args.url)
this._urls = args.url
this._dir = args.dir
this._convert = args.convert
this._format = args.format
this._update = args["update-ffmpeg"]
}
/**
* @param {Array<string>} urls optional - an array of urls
* @param {string} dir? - target directory
* @param {boolean} convert? should the video file be converted
* @param {string} format? format to convert into
* @param {boolean} update? update ffmpeg
*
* Replaces CLI arguments with parameters passed from YtConverter.init funtion
*/
setArgs(urls?: Array<string>, dir?: string, convert?: boolean, format?: string, update?: boolean) {
if (urls)
this._urls = urls
if (dir)
this._dir = dir
if (convert)
this._convert = convert
if (format)
this._format = format
if (update)
this._update = update
if (!this._urls || this._urls.length === 0)
throw new Error("Url argument is required!")
}
get urls() {
return this._urls
}
get dir() {
return this._dir
}
get convert() {
return this._convert
}
get format() {
return this._format
}
get update() {
return this._update
}
}
In the Args.parseCliArgs function I am defining usage, type and the default values for each field. The only required argument will be urls, but I wont add { demand: true } as I want this module to be usable locally. Args.setArgs function will set the values to instance fields, after checking for locally passed parameters.
Next, I will add Instances class that will initialize and hold all the used instances. For now, this class will only initialize Args class and it looks like this:
import Args from "./Args";
export default class Instances {
private _args: Args
constructor() {
this._args = new Args()
}
get args(): Args {
return this._args
}
}
To YtConverter.init function I will add the following lines that will pass the function parameters to this._args instance, check the validity of urls argument, and log the beginning of the process:
async init(urls?: Array<string>, dir?: string, convert?: boolean, format?: string, update?: boolean) {
this._args.setArgs(urls, dir, convert, format, update)
if (!Array.isArray(this._args.urls)) {
throw new Error("The first argument must be an array!")
}
process.stdout.write(`\nDownloading ${this._args.urls.length} videos to directory ${this._args.dir}`)
}
Next, I will add the Downloader class, and install all its required dependencies: npm install youtube-dl @types/youtube-dl humanize. As the last module doesn’t have a type definition module, I’ll have to create it on my own. I’ll create types directory on the root directory and add humanize.d.ts file:
declare module "humanize" {
export function filesize(size: number): string
}
As the filesize function is the only one I’ll use, it will be the only one whose type I’ll define.
The Downloader class will need a few helper functions so I will add a new Utils class with the following content:
import fs from "fs"
import shell from "shelljs"
export default class Utils {
private _interval!: NodeJS.Timeout
ffPath!: string
/**
* @param {string} dir directory path
* Checks if the directory exists, and if not, it creates it.
*/
async checkDir(dir: string) {
return new Promise((resolve) => {
if (!fs.existsSync(dir)) {
shell.mkdir("-p", dir)
resolve()
}
resolve()
})
}
/**
* @param {boolean} run - start of stop loader
*/
loader(run: boolean) {
if (this._interval && !run) {
clearInterval(this._interval)
}
else {
this._interval = setInterval(() => {
process.stdout.write(" .")
}, 200)
}
}
}
The checkDir function will check if the target directory exists, and if it doesn’t, it will create it. The loader function will print dots in the console during longer processes (e.g. video download requests) as a simple UX improvement.
Downloader class now looks like this:
import youtubedl from "youtube-dl";
import fs from "fs"
import path from "path"
import Utils from "./Utils";
import humanize from "humanize"
import ProcessExt from "../types/processExt";
export default class Downloader {
private _utils: Utils
constructor(utils: Utils) {
this._utils = utils
}
/**
* @param {string} url - video url
* - Get video information from youtube
*/
async getVideoInfo(url: string): Promise<youtubedl.Info> {
// check if url is a video url
if (!url.match("/watch?"))
throw new Error("Invalid video url")
return new Promise((res, rej) => {
return youtubedl.getInfo(url, (err: Error, info: any) => {
if (err)
rej(err)
else {
res(info)
}
})
})
}
/**
* @param {string} url - video url
* @param {string} dir - destination directory
* @returns {object} video filename and directory.
*/
async downloadVideos(url: string, dir: string): Promise<{ filename: string, dir: string }> {
this._utils.loader(true)
return new Promise(async (resolve, reject) => {
let processExt: ProcessExt = process
let video: youtubedl.Youtubedl
let videoInfo: youtubedl.Info
let position = 0;
let videoSize = 0;
let filename:string
try {
videoInfo = await this.getVideoInfo(url)
}
catch (err) {
reject(err)
return
}
if (!videoInfo) {
throw new Error(`Unable to get video info from url ${url}`)
}
filename = videoInfo._filename
await this._utils.checkDir(dir)
video = youtubedl(url,
['--format=18'],
{ cwd: dir });
video.on('info', (info: youtubedl.Info) => {
this._utils.loader(false)
videoSize = info.size
console.log(`\nDownloading ${info._filename}`)
console.log(`Size: ${humanize.filesize(info.size)}`);
});
video.on('data', (chunk: any) => {
position += chunk.length;
if (videoSize) {
// calculate video download percentage
var percent = (position / videoSize * 100).toFixed(2);
if (processExt.stdout.cursorTo && processExt.stdout.clearLine) {
processExt.stdout.cursorTo(0);
processExt.stdout.clearLine(1);
process.stdout.write(percent + '%');
}
}
});
video.on('end', async () => {
process.stdout.write(' - DONE\n');
resolve({ filename, dir })
this._utils.loader(false)
})
video.pipe(fs.createWriteStream(`${path.join(dir, filename)}`));
})
}
}
It consists of getVideoInfo and downloadVideos functions. The former fetches the video information using youtubedl module, and the latter downloads the video if the video information was successfully fetched and if the target directory was created.
By calling youtubedl module, video stream is returned that will be piped to the writable stream of a destination file. To the video stream I’ll attach event listeners with handler functions. These functions will simply compare the downloaded size with full video size and log progress in the form of percentages.
In addition, I’ll extend the process.stdout interface to include cursorTo and clearLine functions, as they are not defined in NodeJS.process.stdout type definition by default.
New classes Utils and Downloader will be instantiated in the Instances class, so I’ll add them there:
import Args from "./Args";
import Downloader from "./Downloader";
import Utils from "./Utils";
export default class Instances {
private _args: Args
private _utils: Utils
private _downloader!: Downloader
constructor() {
this._args = new Args()
this._utils = new Utils()
}
get args(): Args {
return this._args
}
get utils(): Utils {
return this._utils
}
get downloader(): Downloader {
if (!this._downloader) {
this._downloader = new Downloader(this.utils)
}
return this._downloader
}
}
I’ll call downloder.downloadVideos function from YtConverter.init function:
for (let url of this._args.urls) {
await this._downloader.downloadVideos(url.toString(), this._args.dir).catch(err => { throw err })
}
Now, download functionality is done, so if I add a call to init function (init()) in index.ts, I will be able to download a video by running:
node ./dist -u [some-yt-video-url]
and by adding -d [destination-dir] I can customize the target directory.
The terminal output of this action looks like this:
The next step is to add a converting functionality. For transcoding video files in other formats, I’ll use ffmpeg, a command-line multimedia software, able to transcode from and to, all the existing media formats. To be able to use ffmpeg, I’ll have to download the binary from its repository using the API endpoint. Additionally, I’ll add an option to update ffmpeg, in the case of new releases.
To start, I’ll add a new directory ffmpeg-downloader with index.ts and ffInfo.json files. The former will hold all the downloading logic and the latter will hold the path to the binary file and the current version number. I’ll start by adding the following content to ffmpeg-downloader/index.ts file:
const LATEST = `http://ffbinaries.com/api/v1/version/latest`
import request, { Response } from "request"
import progress from "request-progress"
import fs from "fs"
import path from "path"
import Utils from "../Utils";
import ffInfo from "./ffInfo.json"
export default class FfmpegDownloader {
utils: Utils
version!: string
url!: string
platform!: string
ffPath!: string
constructor(utils: Utils) {
this.utils = utils
}
async init() {
const { update, version } = await this.checkUpdate()
if (!update) {
console.log(`Ffmpeg already on the latest version: ${version} `)
return
}
await this.downloadVersion()
}
/**
* @returns {object} update: should ffmpeg be updated; version: current ffmpeg version
*
* - Check if ffmpeg should be updated
*/
async checkUpdate(): Promise<{ update: boolean, version: string }> {
const ffVersion = ffInfo.ffVersion
return new Promise((resolve, reject) => {
request.get(LATEST, { json: true }, (err: any, _: Response, body: any) => {
if (err) {
reject(err)
return
}
if (body.version === ffVersion) {
resolve({ update: false, version: ffVersion })
return
}
this.version = body.version
const platform = this.utils.getPlatform()
if (!platform) {
throw new Error("Unable to download ffmpeg")
}
this.url = body.bin[platform].ffmpeg
this.platform = platform
resolve({ update: true, version: this.version })
})
})
}
/**
* @returns Promise<void>
*
* - Downloads ffmpeg.
*/
async downloadVersion():Promise<void> {
const ffDir = path.join(__dirname, `../../ffmpeg/${this.platform}/${this.version}`)
await this.utils.checkDir(ffDir)
this.ffPath = path.join(ffDir, "ffmpeg")
const zipPath = this.ffPath + ".zip"
// If platform is windows, add .exe extension and replace path slashes.
if (this.utils.getPlatform() === "windows-64") {
this.ffPath += ".exe"
this.ffPath = this.ffPath.replace(/\\/g, "/")
}
fs.writeFileSync(path.join(__dirname, "./ffInfo.json"), Buffer.from(`{ "ffPath": "${this.ffPath}","ffVersion":"${this.version}"}`))
// Download ffmpeg
return new Promise((resolve, reject) => {
process.stdout.write(`Downloading ffmpeg v${this.version}`)
this.utils.loader(true)
progress(request.get(this.url, { json: true }))
.on("end", async () => {
this.utils.loader(false)
process.stdout.write(`\nExtracting ffmpeg v${this.version}`)
await this.utils.extract(zipPath, ffDir)
resolve()
})
.on("error", () => {
reject("An error occured")
return
})
.pipe(fs.createWriteStream(zipPath))
})
}
}
For this to work, I have to install additional dependencies: npm install request request-progress @types/request and add a type definition for request-progress module in types/requestProgress.d.ts file:
declare module "request-progress"
FfmpegDownloader class uses 2 additional helper functions: getPlatform and extract, so I’ll add them to the Utils class. getPlatform function checks for the system platform using Node.js “os” module so the correct binary could be downloaded. As the binary is downloaded in the .zip format, extract extracts the file in the same directory, and deletes the original file.
/**
* @param {string} path target file path
* @param {string} dir destination directory
* @returns Promise
*/
async extract(path: string, dir: string): Promise<void> {
return new Promise((resolve, reject) => {
this.loader(true)
extract(path, { dir: dir, defaultFileMode: 777 }, (err: Error | undefined) => {
if (err) {
reject(err)
this.loader(false)
return
}
fs.unlink(path, (err: NodeJS.ErrnoException | null) => {
if (err) {
reject(err)
return
}
resolve()
this.loader(false)
})
})
})
}
/**
* @returns string | null
*/
getPlatform(): string | null {
switch (os.platform()) {
case ("win32"): {
return "windows-64"
}
case ("linux"): {
if (os.arch() === "x64")
return "linux-64"
else if (os.arch() === "x32")
return "linux-32"
}
case ("darwin"): {
return "osx-64"
}
default: return null
}
}
Now ffmpeg downloading functionality is done. If I run: npm run build && node dist/index.js -u https://www.youtube.com/watch\?v=zMqP1sIheeM -d ./video --uf , ffmpeg will be downloaded before file download. If I run the command again, the download will be skipped as the versions are the same.
The last thing left to do is the use the ffmpeg binary to transcode the downloaded videos. To accomplish this, I’ll add a new Converter class in Converter.ts file:
import fs from "fs"
import path from "path"
import Utils from "./Utils";
import { exec } from "child_process"
import ProcessExt from "../types/processExt";
import { ExecFileError } from "../types/execFileError";
export default class Converter {
private _utils: Utils
constructor(utils: Utils) {
this._utils = utils
}
/**
* @param {string} file - filename to convert
* @param {string} dir - destination directory
* @param {string} format - format to convert into
* - Converts a file into a given format.
*/
async convert(file: string, dir: string, format: string) {
const fileArr = file.split(".")
let ext = fileArr.pop() || "mp4"
// Read ffmpeg binary path from ffInfo.json
const ffPath = JSON.parse(fs.readFileSync(path.join(__dirname, "./ffmpeg-downloader/ffInfo.json")).toString()).ffPath
const filename = fileArr.join(".")
console.log(`\nConverting ${file} to ${filename}.${format}`)
await this._utils.checkDir(`${dir}/${format}`)
let cmd = `${ffPath} -y -i ${dir}/"${filename.concat(".", ext)}" ${dir}/"${format}/${filename}.${format}"`
return new Promise((resolve) => {
let duration = 0
const converter = exec(cmd,
(err: ExecFileError | null) => {
if (err) {
if (err.code && err.code === "ENOENT") {
throw new Error("ffmpeg must be installed!")
}
throw err
}
})
if (!converter.stderr) {
throw new Error("Invalid commmand.")
}
converter.stderr.on("end", () => {
resolve()
})
converter.stderr.on("data", (data: any) => {
const durationIndex = data.search("Duration")
if (durationIndex !== -1) {
// find the duration substring in ffmpeg output
let durData = data.substring(durationIndex, durationIndex + 18).split(" ")[1].trim()
duration = this._utils.getDuration(durData)
}
else {
let processE: ProcessExt = process
const timeIndex = data.search("time=")
if (timeIndex === -1)
return
const timeData = data.substring(timeIndex, timeIndex + 13).split("=")[1].trim()
const time = this._utils.getDuration(timeData)
if (duration === NaN || time === NaN)
return
const percentage = ((time / duration) * 100) <= 100 ? ((time / duration) * 100).toFixed(2) : "100.00"
if (processE.stdout.cursorTo && processE.stdout.clearLine) {
processE.stdout.cursorTo(0);
processE.stdout.clearLine(1);
processE.stdout.write(percentage + "%");
}
if (percentage == "100.00")
process.stdout.write(" - DONE\n");
}
})
})
}
}
For this to work, I’ll need to add getDuration function to Utils class:
/**
* @param {string} data string in format hh:mm:ss
* @returns {number} duration in miliseconds
*/
getDuration(data: string): number {
let dataArr = data.split(":")
let duration = 0
if (dataArr.length !== 3)
return NaN
dataArr.forEach((d: string, i: number) => {
switch (i) {
case (0): duration += parseInt(d) * 3600 * 1000
break;
case (1): duration += parseInt(d) * 60 * 1000
break;
case (2): duration += parseInt(d) * 1000
}
})
return duration
}
This function will attempt to parse duration string from ffmpeg output in the format [hh:mm:ss] and return the duration in miliseconds. This data will be used to generate progress percentage that will be printed to the console.
Converter class consists of convert function that reads ffmpeg path and executes the binary with arguments provided through the command-line or as function parameters. exec function returns a child_process object that can be listened for events. I’m listening for “data” events to parse the duration string and log the progress percentage and the “end” event to resolve the Promise.
Next, I’ll add the converter to instances, import it in YtConverter class and run the converter if the convert argument is true:
for (let url of this._args.urls) {
const { filename, dir } = await this._downloader.downloadVideos(url.toString(), this._args.dir).catch(err => { throw err })
if (this._args.convert)
await this._converter.convert(filename, dir, this._args.format).catch((err: Error) => { throw err })
}
All that is left to do now is to prepare this module for global installation.
Preparing for global installation
Create a new folder named bin inside the root directory of your module with an exec.js file in it with the following content:
#!/usr/bin/env node
require('../dist/index.js').init()
The first line (#!/usr/bin/env node) will allow npm to correctly generate an executable for this module.
The last thing to do is to register the package as global by adding the following to package.json:
"bin": {
"ytc": "./bin/exec.js"
}
Here, I’m setting the path to the executable file which will be run with ytc command. Now this module can be installed with the -g option. You can test this by running in the root directory:
npm install -g .
The module is now ready. I’ll make an additional modification that will run ffmpegDownloader after the module is installed: I’ll add a small script in src/scripts/downloadFfmpeg.js with the following content:
import Instances from "../Instances"; const instances = new Instances() instances.ffmpegDownloader.init()
For the script to run before the installation, I’ll add “postinstall” script in package.json.
"postinstall": "tsc && node ./dist/scripts/downloadFFmpeg.js",
After the installation, this module can be run from the command-line:
ytc -u [video-url] -d [custom-directory] -c
For the list of all options, run:
ytc --help
The code for this project can be found at https://github.com/learningdollars/martas-youtubedownloader.
Hello,
This is a nice article. Please let me know what are the dependencies/other modules used in the node/npm module ? Are there any specific version constriants?
Hi, all the dependencies and their versions are listed in package.json. All you have to do is to run npm install. Did you manage to run this?
Yes, it worked. What can be done to make the requests thru a proxy and do throttling?
Keep posting such helpful articles, easy for intermediate level developers to follow along. Add one-liner comments in some places; it will help anyone to understand the material better. Keep up the excellent work!
Yes it worked.How to throttle the requests and send them using proxy in between ,please suggest.
Yes,it worked.How to throttle the requests and fire them thru proxy to avoid blocking?
Thanks for this article. It is incredible to see highly motivated developers like you making things easy for others by writing such helpful blogs.
Future Use Case:
I was thinking that if you get enough node modules for different websites like facebook, Vimeo, MetaCafe, Dailymotion, then you can make a universal video downloader/converter by mixing them using a template pattern and provide a unified interface that identifies the source of video using regex and then use the design pattern to plug in the suitable class function.
You’ve added your code as blocks, not a screenshot, that is comfortable for anyone who wants to follow along and try things out,
Overall a balanced article with the right level of details and implementation explanation.
Social media and digital media are both in style. Users are growing every day, and since every social media platform wants to increase views and subscriptions, they have made it impossible for users to download anything, allowing them to return repeatedly and reuse the same material. In order to download, you should bookmark this popular social media Pinterest downloader. Verify that and save the link. https://pinvideodown.com/
I want to develop these tools this article is very help for me.Thank you.