Batch Files: Shell Scripting For Even File Distribution

Aug 7, 2025 by ADMIN 56 views

Creating N Folders with Equal File Distribution Using Shell Scripting

Hey guys! Ever found yourself drowning in a sea of files in one massive folder and wished you could neatly organize them into batches? Well, you're not alone! Many of us, especially those working with large datasets or media files, often need to split them into subfolders for easier management, processing, or distribution. If you're like me and not a bash scripting guru just yet, don't worry! This article will walk you through a simple yet effective shell script to create 'n' folders and evenly distribute your files among them. Let's dive in and get those files organized!

Why Batch Files?

Before we jump into the how-to, let's quickly touch on the why. Batching files is super useful in various scenarios. Think about it: when dealing with thousands (or even millions!) of files, it's often impractical to process them all at once. Splitting them into smaller, manageable chunks makes things way easier. For example, if you're training a machine learning model, you might want to divide your data into batches to feed into the model iteratively. Or, if you're backing up files to different storage locations, batching can help streamline the process. Plus, it just makes your file system cleaner and more organized, which is a win in itself!

Benefits of Using Shell Script for File Batching

Shell scripting offers a powerful and flexible way to automate tasks like file batching. Here are some key advantages:

Automation: Shell scripts allow you to automate repetitive tasks, saving you time and effort. Instead of manually creating folders and moving files, a script can do it all for you with a single command.
Flexibility: You can customize your script to fit your specific needs. Want to specify the number of batches? No problem! Need to handle different file types or sizes? A script can be modified to accommodate various requirements.
Efficiency: Shell scripts are generally very efficient, especially for file system operations. They can quickly process large numbers of files and folders.
Portability: Shell scripts are platform-independent, meaning they can run on any Unix-like operating system (Linux, macOS, etc.).

Prerequisites

Before we get started with the script, make sure you have the following:

A Unix-like operating system: This tutorial assumes you're using a Unix-like system (Linux, macOS, etc.) with a bash shell.
Basic familiarity with the command line: You should be comfortable navigating directories, listing files, and running basic commands in the terminal.
A directory with files to batch: Have the directory containing the files you want to batch ready.

Step-by-Step Guide to Creating the Script

Okay, let's get our hands dirty and write the script! We'll break it down step-by-step so it's easy to follow.

Step 1: Setting Up the Script

First, create a new file using your favorite text editor (like nano, vim, or gedit). Let's name it batch_files.sh. Add the following shebang line at the top of the file. This tells the system to execute the script using bash:

#!/bin/bash

Next, we'll add some comments to explain what the script does. This is good practice for making your script more readable and maintainable:

#!/bin/bash

# This script creates n folders and distributes files evenly among them.

Step 2: Getting User Input

We want our script to be flexible, so let's allow the user to specify the number of folders they want to create and the directory containing the files. We'll use the read command to get this input:

#!/bin/bash

# This script creates n folders and distributes files evenly among them.

# Prompt the user for the number of folders to create
read -p "Enter the number of folders to create: " num_folders

# Prompt the user for the directory containing the files
read -p "Enter the directory containing the files: " source_dir

Here, -p option allows us to display a prompt to the user. The input is then stored in the variables num_folders and source_dir.

Step 3: Basic Input Validation

It's always a good idea to validate user input to prevent errors. Let's check if the number of folders is a positive integer and if the source directory exists:

#!/bin/bash

# This script creates n folders and distributes files evenly among them.

# Prompt the user for the number of folders to create
read -p "Enter the number of folders to create: " num_folders

# Prompt the user for the directory containing the files
read -p "Enter the directory containing the files: " source_dir

# Validate input
if ! [[ "$num_folders" =~ ^[0-9]+$ ]] || [ "$num_folders" -le 0 ]; then
  echo "Error: Invalid number of folders. Please enter a positive integer."
  exit 1
fi

if ! [ -d "$source_dir" ]; then
  echo "Error: Directory '$source_dir' does not exist."
  exit 1
fi

This code checks if num_folders contains only digits and is greater than 0. It also verifies if the directory specified by source_dir exists using the -d option. If either check fails, an error message is displayed, and the script exits.

Step 4: Creating the Folders

Now, let's create the folders. We'll use a loop to create folders named batch_1, batch_2, and so on:

#!/bin/bash

# This script creates n folders and distributes files evenly among them.

# Prompt the user for the number of folders to create
read -p "Enter the number of folders to create: " num_folders

# Prompt the user for the directory containing the files
read -p "Enter the directory containing the files: " source_dir

# Validate input
if ! [[ "$num_folders" =~ ^[0-9]+$ ]] || [ "$num_folders" -le 0 ]; then
  echo "Error: Invalid number of folders. Please enter a positive integer."
  exit 1
fi

if ! [ -d "$source_dir" ]; then
  echo "Error: Directory '$source_dir' does not exist."
  exit 1
fi

# Create the folders
for ((i=1; i<=$num_folders; i++))
do
  mkdir -p "${source_dir}/batch_$i"
done

The for loop iterates from 1 to num_folders. Inside the loop, mkdir -p creates the folders. The -p option ensures that the command doesn't fail if the folder already exists.

Step 5: Distributing the Files

This is the core part of the script. We'll get a list of files, calculate how many files should go into each folder, and then move them accordingly. First, let's get the list of files:

#!/bin/bash

# This script creates n folders and distributes files evenly among them.

# Prompt the user for the number of folders to create
read -p "Enter the number of folders to create: " num_folders

# Prompt the user for the directory containing the files
read -p "Enter the directory containing the files: " source_dir

# Validate input
if ! [[ "$num_folders" =~ ^[0-9]+$ ]] || [ "$num_folders" -le 0 ]; then
  echo "Error: Invalid number of folders. Please enter a positive integer."
  exit 1
fi

if ! [ -d "$source_dir" ]; then
  echo "Error: Directory '$source_dir' does not exist."
  exit 1
fi

# Create the folders
for ((i=1; i<=$num_folders; i++))
do
  mkdir -p "${source_dir}/batch_$i"
done

# Get the list of files
files=$(find "$source_dir" -maxdepth 1 -type f -print0 | sort -z)
num_files=$(echo -n "$files" | wc -l)

Here, find command lists all files in the source_dir (using -maxdepth 1 to avoid subdirectories and -type f to only list files). The -print0 option and sort -z ensure that filenames with spaces or special characters are handled correctly. We then count the number of files using wc -l.

Next, let's calculate the number of files per folder and distribute them:

#!/bin/bash

# This script creates n folders and distributes files evenly among them.

# Prompt the user for the number of folders to create
read -p "Enter the number of folders to create: " num_folders

# Prompt the user for the directory containing the files
read -p "Enter the directory containing the files: " source_dir

# Validate input
if ! [[ "$num_folders" =~ ^[0-9]+$ ]] || [ "$num_folders" -le 0 ]; then
  echo "Error: Invalid number of folders. Please enter a positive integer."
  exit 1
fi

if ! [ -d "$source_dir" ]; then
  echo "Error: Directory '$source_dir' does not exist."
  exit 1
fi

# Create the folders
for ((i=1; i<=$num_folders; i++))
do
  mkdir -p "${source_dir}/batch_$i"
done

# Get the list of files
files=$(find "$source_dir" -maxdepth 1 -type f -print0 | sort -z)
num_files=$(echo -n "$files" | wc -l)

# Distribute the files
files_per_folder=$((num_files / num_folders))
remainder=$((num_files % num_folders))

file_index=0
for ((i=1; i<=$num_folders; i++))
do
  folder_files=$files_per_folder
  if [ "$i" -le "$remainder" ]; then
    folder_files=$((folder_files + 1))
  fi

  for ((j=1; j<=$folder_files; j++))
do
    filename=$(echo "$files" | head -n $((file_index + 1)) | tail -n 1)
    mv "$filename" "${source_dir}/batch_$i/"
    file_index=$((file_index + 1))
  done
done

We calculate files_per_folder and remainder to handle cases where the files cannot be divided equally. The outer loop iterates through each folder, and the inner loop moves the appropriate number of files into the folder. If there's a remainder, the first few folders get an extra file each.

Step 6: Finishing Touches

Let's add a message to let the user know the script is done:

#!/bin/bash

# This script creates n folders and distributes files evenly among them.

# Prompt the user for the number of folders to create
read -p "Enter the number of folders to create: " num_folders

# Prompt the user for the directory containing the files
read -p "Enter the directory containing the files: " source_dir

# Validate input
if ! [[ "$num_folders" =~ ^[0-9]+$ ]] || [ "$num_folders" -le 0 ]; then
  echo "Error: Invalid number of folders. Please enter a positive integer."
  exit 1
fi

if ! [ -d "$source_dir" ]; then
  echo "Error: Directory '$source_dir' does not exist."
  exit 1
fi

# Create the folders
for ((i=1; i<=$num_folders; i++))
do
  mkdir -p "${source_dir}/batch_$i"
done

# Get the list of files
files=$(find "$source_dir" -maxdepth 1 -type f -print0 | sort -z)
num_files=$(echo -n "$files" | wc -l)

# Distribute the files
files_per_folder=$((num_files / num_folders))
remainder=$((num_files % num_folders))

file_index=0
for ((i=1; i<=$num_folders; i++))
do
  folder_files=$files_per_folder
  if [ "$i" -le "$remainder" ]; then
    folder_files=$((folder_files + 1))
  fi

  for ((j=1; j<=$folder_files; j++))
do
    filename=$(echo "$files" | head -n $((file_index + 1)) | tail -n 1)
    mv "$filename" "${source_dir}/batch_$i/"
    file_index=$((file_index + 1))
  done
done

echo "Files distributed into $num_folders folders."

Running the Script

Make the script executable:
```
chmod +x batch_files.sh
```
Run the script:
```
./batch_files.sh
```
The script will prompt you for the number of folders and the directory containing the files. Enter the values, and the script will do its magic!

Complete Script

Here's the complete script for your reference:

#!/bin/bash

# This script creates n folders and distributes files evenly among them.

# Prompt the user for the number of folders to create
read -p "Enter the number of folders to create: " num_folders

# Prompt the user for the directory containing the files
read -p "Enter the directory containing the files: " source_dir

# Validate input
if ! [[ "$num_folders" =~ ^[0-9]+$ ]] || [ "$num_folders" -le 0 ]; then
  echo "Error: Invalid number of folders. Please enter a positive integer."
  exit 1
fi

if ! [ -d "$source_dir" ]; then
  echo "Error: Directory '$source_dir' does not exist."
  exit 1
fi

# Create the folders
for ((i=1; i<=$num_folders; i++))
do
  mkdir -p "${source_dir}/batch_$i"
done

# Get the list of files
files=$(find "$source_dir" -maxdepth 1 -type f -print0 | sort -z)
num_files=$(echo -n "$files" | wc -l)

# Distribute the files
files_per_folder=$((num_files / num_folders))
remainder=$((num_files % num_folders))

file_index=0
for ((i=1; i<=$num_folders; i++))
do
  folder_files=$files_per_folder
  if [ "$i" -le "$remainder" ]; then
    folder_files=$((folder_files + 1))
  fi

  for ((j=1; j<=$folder_files; j++))
do
    filename=$(echo "$files" | head -n $((file_index + 1)) | tail -n 1)
    mv "$filename" "${source_dir}/batch_$i/"
    file_index=$((file_index + 1))
  done
done

echo "Files distributed into $num_folders folders."

Conclusion

And there you have it! A simple shell script to create 'n' folders and evenly distribute your files. This can be a huge time-saver when dealing with large amounts of data. Feel free to customize the script further to suit your specific needs. Happy scripting, folks!

FAQs

What if I want to copy the files instead of moving them?

Change the mv command to cp -r in the script. This will copy the files instead of moving them.

How can I handle different file types?

You can modify the find command to filter files based on their extension. For example, to only include .txt files, use find "$source_dir" -maxdepth 1 -type f -name "*.txt" -print0.

Can I specify a custom folder name prefix?

Yes! Modify the mkdir command and the destination path in the mv command to include your desired prefix.

What if the number of files is less than the number of folders?

The script will still work, but some folders may end up being empty. You can add additional logic to handle this case if needed.