Limiting Wordpress memory usage when doing a bulk insert of Wordpress

Sometimes you need to bulk import lots of posts from different sources into Wordpress. This is quite a simple process using the Wordpress API:

<?php

ini_set('memory_limit','600M');

require_once('../../../wp-load.php');

$count = 1;

$to_add = 50000;

echo "Adding $to_add posts..." . PHP_EOL;

while(true) {
    if($count>$to_add)
        break;
    
    // Create post object
    $my_post = array(
      'post_title'    => 'My post ' . $count,
      'post_content'  => 'This is my post.',
      'post_status'   => 'publish',
      'post_author'   => 1,
      'post_category' => array(8,39)
    );
    
    // Insert the post into the database
    wp_insert_post( $my_post );
    
    $count++;
}

echo "Done." . PHP_EOL;

echo "Peak memory used (MB): ". memory_get_peak_usage() / (1024*1024)  . PHP_EOL;

?>

The above example works well for a small number of posts (somewhere around 10,000 posts); however when you have to import a larger number of posts, the above code ends up taking alot of memory, and can cause your server to run out of memory. For example, the above code (50,000 posts added) took up 209 MB of memory when importing into a new instance of Wordpress (no plugins).

Depending on which plugins you have enabled the amount of memory the above script takes can vary considerably. Running the above script with lots of other plugins could cause it to take even more that 209 MB.

One thing you can do to make the above script use less memory is to turn off the Wordpress cache. This can be done by calling the "wp_suspend_cache_addition" function:

<?php
ini_set('memory_limit','600M');

require_once('../../../wp-load.php');

wp_suspend_cache_addition(true);

$count = 1;
$to_add = 50000;
echo "Adding $to_add posts..." . PHP_EOL;

while(true) {
    if($count>$to_add)
        break;
    
    // Create post object
    $my_post = array(
      'post_title'    => 'My post ' . $count,
      'post_content'  => 'This is my post.',
      'post_status'   => 'publish',
      'post_author'   => 1,
      'post_category' => array(8,39)
    );
    
    // Insert the post into the database
    wp_insert_post( $my_post );
    
    $count++;
}

echo "Done." . PHP_EOL;

echo "Peak memory used (MB): ". memory_get_peak_usage() / (1024*1024)  . PHP_EOL;

?>

After the addtion of the "wp_suspend_cache_addition" function the above script - adding 50,000 posts takes up only 14 MB - thats a reduction of 93%.

The above script was tested on Wordpress 3.6.1.

Making Wordpress's search form submit to a 'friendly' URL

In Wordpress, you can customize how URLs are structured, via the permalinks page.

So, for example posts that looked like

http://example.com/?p=1

can be configured to look like

http://example.com/hello-world/

The second URL is obviously easier to read, more aesthetically pleasing and of course, more SEO friendly, as now the URL itself is giving us a hint about the content of the page.

This is done simply making some updates in the permalinks portion of the wp-admin page.

Wordpress 5.2's permalinks page.

While this works great for posts, you will notice that this does not extend to the search form, which by default still submits using a query string:

http://example.com/?s=hello

Interestingly, you will find that after you have changed your permalink setting, a URL of the format /search/{search term} does exist, so if you enter:

http://example.com/search/hello

you will be taken to the search page.

However, a simple HTML form can't submit to this friendly search URL, as the URL itself contains the search term. (A the form's action attribute will need a complete URL, including the search term - but we can't know the search term when defining the form).

The most straight forward workaround for this is add a redirect in Wordpress to make sure the '?s=hello' search page is redirected to '/search/hello' instead. An example this can be found here: http://wpengineer.com/2258/change-the-search-url-of-wordpress/

Another option is to use Javascript and change the form's action attribute just before submitting the form to the server.

Here is the code:

<script type="text/javascript">
    function doSearchSubmit(searchForm) {
        
        //get the search form's input field
        var s = jQuery(searchForm).find('#s'); 
        
        //change the form's action attribute to include the search term
        searchForm.action='<?php bloginfo('url'); ?>/search/'+s.val(); 
        
        //disable the form's input field so that it is not submitted with the form
        s.attr('disabled', 'disabled');
        
        return true;        
    }
</script>

<form role="search" method="get" id="searchform" action="<?php bloginfo('url'); ?>" onsubmit="return doSearchSubmit(this);"> 
 <div><label class="screen-reader-text" for="s"><?php _e('Search for:'); ?></label> 
 <input type="text" value="" name="s" id="s" /> 
 <input type="submit" id="searchsubmit" value="<?php _e('Search') ?>" /> 
 </div> 
</form>

The above code goes in 'searchform.php' which should be at the root of your theme. If there isn't one there already, you can add one.

It's pretty simple - instead of submitted the from directly, we use some Javascript to change the action attribute of the from to the friendly URL instead.

This works as expected however there is one minute issue I noticed while testing on Chrome: when you submit this form the URL it goes to looks like this:

http://example.com/search/hello?

(This doesn't happen on Firefox 22.0 or IE 9).

To overcome this issue on Chrome, we can set the form method attribute to 'post' instead of 'get' - this will ensure no query string remnants are seen on any browser.

Another thing we could consider is to never have the form submit at all - instead we just change the location of the page to go directly to the friendly search URL:

<script type="text/javascript">
 function doSearchSubmit(searchForm) {

        //get the search form's input field
        var s = jQuery(searchForm).find('#s');         
        
        //just redirect to the search page
        window.location='<?php bloginfo('url'); ?>/search/'+s.val(); 
        
        return false;
    }
</script>

<form role="search" method="get" id="searchform" action="<?php bloginfo('url'); ?>" onsubmit="return doSearchSubmit(this);"> 
 <div><label class="screen-reader-text" for="s"><?php _e('Search for:'); ?></label> 
 <input type="text" value="" name="s" id="s" /> 
 <input type="submit" id="searchsubmit" value="<?php _e('Search') ?>" /> 
 </div> 
</form>