Limiting Wordpress memory usage when doing a bulk insert of Wordpress

Sometimes you need to bulk import lots of posts from different sources into Wordpress. This is quite a simple process using the Wordpress API:

<?php

ini_set('memory_limit','600M');

require_once('../../../wp-load.php');

$count = 1;

$to_add = 50000;

echo "Adding $to_add posts..." . PHP_EOL;

while(true) {
    if($count>$to_add)
        break;
    
    // Create post object
    $my_post = array(
      'post_title'    => 'My post ' . $count,
      'post_content'  => 'This is my post.',
      'post_status'   => 'publish',
      'post_author'   => 1,
      'post_category' => array(8,39)
    );
    
    // Insert the post into the database
    wp_insert_post( $my_post );
    
    $count++;
}

echo "Done." . PHP_EOL;

echo "Peak memory used (MB): ". memory_get_peak_usage() / (1024*1024)  . PHP_EOL;

?>

The above example works well for a small number of posts (somewhere around 10,000 posts); however when you have to import a larger number of posts, the above code ends up taking alot of memory, and can cause your server to run out of memory. For example, the above code (50,000 posts added) took up 209 MB of memory when importing into a new instance of Wordpress (no plugins).

Depending on which plugins you have enabled the amount of memory the above script takes can vary considerably. Running the above script with lots of other plugins could cause it to take even more that 209 MB.

One thing you can do to make the above script use less memory is to turn off the Wordpress cache. This can be done by calling the "wp_suspend_cache_addition" function:

<?php
ini_set('memory_limit','600M');

require_once('../../../wp-load.php');

wp_suspend_cache_addition(true);

$count = 1;
$to_add = 50000;
echo "Adding $to_add posts..." . PHP_EOL;

while(true) {
    if($count>$to_add)
        break;
    
    // Create post object
    $my_post = array(
      'post_title'    => 'My post ' . $count,
      'post_content'  => 'This is my post.',
      'post_status'   => 'publish',
      'post_author'   => 1,
      'post_category' => array(8,39)
    );
    
    // Insert the post into the database
    wp_insert_post( $my_post );
    
    $count++;
}

echo "Done." . PHP_EOL;

echo "Peak memory used (MB): ". memory_get_peak_usage() / (1024*1024)  . PHP_EOL;

?>

After the addtion of the "wp_suspend_cache_addition" function the above script - adding 50,000 posts takes up only 14 MB - thats a reduction of 93%.

The above script was tested on Wordpress 3.6.1.