Creating a Magento Dev Site

As any web-developer knows a development site is a must have for any major production site, as you shouldn’t just go around making changes to a live (working) site. If you do, there is a real chance that actual visitors will encounter a bug/problem while browsing the site.

This can be a bad thing for any website, but is a much bigger deal with an ecommerce site, where if a user tries and is unable to successfully use the website, they are likely going to close their browser tab or hit back and go find a different supplier. As a smaller(read non-amazon level seller) ecommerce site, this often isn’t just a single lost sale, but also possible future sales as well.

Of course, downtime can be a problem with any sort of website, including more oriented service, as opposed to product based, websites, so it is best practices to work on a development site whenever possible. However, it is often much easier to quantify the direct monetary impact that downtime can have on an ecommerce website, so there is rarely a good excuse for making live changes to a production ecommerce website.

This post covers how to make a clone of an exiting Magneto Eccomerce site, so that you can make changes to the site template, plugins, or other features, without impacting real customers.

Prerequisites

This article assumes that you have a working web server with two websites setup. For the purpose of this example, they will use the url store.example.com, which is a working Magento website, and dev.example.com, which is a new empty website. These sites should be accessible via your web-browser and you should have file and database access to the server. You should also be familiar with working with SSH to access your web server and database.

Store Variables for the Purpose of This Tutorial

• Live Store URL: store.example.com
• Dev Store URL: dev.example.com
• Live Store Database: magento_live
• Dev Store Database: magento_dev
• Live Store Web Root: /home/magento/public_html
• Dev Store Web Root: /home/mangento/public_html_dev

Basic Overview

Cloning a Magento store requires a few steps of preparation and some waiting(depending on how big your Magento Store is,) but is relatively straightforward.

The steps are as follows:

1. Make a Copy the Live Store’s Files
2. Give the Development Store Permissions a Sanity Check
3. Make a Copy of the Live Stores Database
4. Update the Development Database to use the New Store URL
5. Update the Development Config Files to use the New Database
6. Manually Clear the Development Stores Cache
7. Wrapping Up and Other Considerations

Make a Copy the Live Store’s Files

First, you will need to copy over all the Magento files from the live website into the new development websites webroot. Before executing the following command, ensure that the new development webroot is empty.

cp -rf /home/magento/public_html /home/magento/public_html_dev

The above will copy files and folders from your live Magento Webroot(/home/magento/public_html) into your development Magento Webroot(/home/magento/public_html_dev).

Back to Table of Contents

Give the Development Store Permissions a Sanity Check

Magento is easiest to setup when the webserver has access to all the Magento files. Of course, there are some security implications to this and a strong argument can be made for limiting what files the webserver has write access to. However, advanced securing of the Magento install is out of the scope of this tutorial, so we will go with the path of least resistance when setting up the development website and assume that your webserver has full control over the Magento files.

The below commands ensure that the newly copied files all have the correct permissions. This step may not be necessary, but not having the incorrect permissions can cause problems, so it is good to give it a sanity check.

cd /home/magento/public_html_dev
find . -type f -exec chmod 644 {} \;
find . -type d -exec chmod 755 {} \;
chmod +x mage

Back to Table of Contents

Make a Copy of the Live Stores Database

Next, we need to make a copy of the live website’s database and import it into the new development database. Like securing Magento, it is possible to do this in a single command, but for simplicity sake, we will break this up into two parts: Making a copy of the live database and importing the live database into the development database

First, lets use mysqldump to make a backup of the live database:

mysqldump -u root -p magento_live > ~/magento_live_2016-06-20.sql

Next, we will copy the live database into our development database:

mysql -u root -p magento_dev < ~/magento_live_2016-06-20.sql

The above command will import the live database dump file into our new database, so that any changes we make while working on the development site do not impact the live store website.

Depending on the size of your store, this could take quite some time to complete, often longer than the initial dump, so be patient. If you are on a server with limited resources, you may wish to do this step during the evening or an off time, so that it doesn’t impact your live website.

Back to Table of Contents

Update the Development Database to use the New Store URL

Like most database driven E-Commerce Platforms or Content Management Systems, many of the configuration settings are stored in the database. So, we can’t just start using the database without updating a couple settings, specifically the stores URL. Currently, it is still set to the live website url(store.example.com,) so before we can use it, we need to change the store url to the development url(dev.example.com.) Since we can’t access the site admin to do this, we will need to login to the database and update it there.

First, lets login and look at the current store url:

mysql -u root -p magento_dev
MariaDB [magento_dev]> select * from core_config_data where path='web/unsecure/base_url' or path='web/secure/base_url';
+-----------+---------+----------+-----------------------+-----------------------------+
| config_id | scope   | scope_id | path                  | value                       |
+-----------+---------+----------+-----------------------+-----------------------------+
|         6 | default |        0 | web/unsecure/base_url | http://store.example.com/   |
|         7 | default |        0 | web/secure/base_url   | https://store.example.com/  |
+-----------+---------+----------+-----------------------+-----------------------------+

The above command shows the current values for the web/unsecure/base_url and web/secure/base_url config settings. Alternatively, you can also preform the below query to find all settings that contain the store’s current URL. This can be useful if you have other plugins/modules installed that contain the store’s url or if you modified other settings, such as web/cookie/cookie_domain. Go ahead and run it to be sure and review the result’s value column, but for most installs, you will probably only need to adjust the base_url fields.

MariaDB [magento_dev]> select * from core_config_data where value like '%store.example.com%';

Next, we need to update the settings to use the new url. For the purpose of this tutorial, we will assume that you have an SSL certificate setup for the development store. If you do not, you should, but you could also just use the non-ssl version of the url for the ‘web/secure/base_url’. Especially if you choose to use the non-ssl version, you should also update the password of your admin user(s,) so that they do not login over an insecure session using their live Magento password.

MariaDB [magento_dev]> update core_config_data set value='http://dev.example.com/' where path = 'web/unsecure/base_url' limit 1;
MariaDB [magento_dev]> update core_config_data set value='https://dev.example.com/' where path = 'web/secure/base_url' limit 1;

Now, lets check one more time to make sure the urls are correct:

MariaDB [magento_dev]> select * from core_config_data where path='web/unsecure/base_url' or path='web/secure/base_url';
+-----------+---------+----------+-----------------------+---------------------------+
| config_id | scope   | scope_id | path                  | value                     |
+-----------+---------+----------+-----------------------+---------------------------+
|         6 | default |        0 | web/unsecure/base_url | http://dev.example.com/   |
|         7 | default |        0 | web/secure/base_url   | https://dev.example.com/  |
+-----------+---------+----------+-----------------------+---------------------------+

Back to Table of Contents

Update the Development Config Files to use the New Database

Since we just directly copied over all the files from the live website, we need to update the database configuration file to use the new database before we attempt to access the site. If we don’t, the development site will pull the site configuration from the live website database, which will cause problems.

The database configuration file is typically located at: /home/magento/public_html_dev/app/etc/local.xml

Using your favorite text editor(VIM,) open the file and edit the connection section of the XML configuration file:

            <default_setup>
                <connection>
                    [...]
                    <username><![CDATA[your_database_user]]></username>
                    <password><![CDATA[your_database_password]]></password>
                    <dbname><![CDATA[magento_dev]]></dbname>
                    [...]                    
                </connection>
            </default_setup>

At minimum, you need to change the dbname variable to be the new development database, so for the purpose of this tutorial magento_dev. However, depending on your setup, you may also need to adjust the username/password as well.

You will note that Magento encloses all of the XML values within a <![CDATA[ ]]> block. This is simply a way to ensure that characters that might otherwise be interpreted as part of the XML, like a less then sign(<) or greater then sign(>) do not cause the XML to break. You will need to edit the part in between the brackets: []

After making the above change to the dbname variable, make sure to save the file and close your text editor.

Back to Table of Contents

Manually Clear the Development Stores Cache

Most likely, if you are running a live Magento website, you are making use of their built-in caching system. As a last step, lets delete the cache and sessions so that all logins are cleared and any old cached data is removed.

rm -rf /home/magento/public_html_dev/var/cache/*
rm -rf /home/magento/public_html_dev/var/session/*

The above should clear out your cache. Note that during development, it is often useful to disable the cache all together, especially if you are editing the template or modifying/installing Magento plugins, modules, or code.

Back to Table of Contents

Wrapping Up and Other Considerations

The above steps should get you a working development site. You should now be able to visit dev.example.com and it should look and behave similar to your live Magento store. It should also be possible to login to the Magento Admin using your current login.

You will probably want to take a few additional steps to ensure that the development site is not visible to search engines.

Adding a deny all to your sites robots.txt file is a good first step, as shown below. However, keep in mind that even with a robots.txt, bots will still crawl your sites.

Even google doesn’t seem to 100% respect your robots.txt or a meta noindex, nofollow tag, so I typically get a bit more aggressive and use .htaccess file to block some of the more common bots, like Google and MSN. It won’t catch them all, but really your main concern is keeping Google/Bing from indexing your development website, so it is likely sufficient.

First, create a robots.txt file(if one doesn’t exist) and add the following:

User-agent: *
Disallow: /

Next, edit your .htaccess to block some of the more common bots:

BrowserMatchNoCase bot probably_a_bot
BrowserMatchNoCase spider probably_a_bot
BrowserMatchNoCase crawler probably_a_bot
BrowserMatchNoCase rogerbot probably_a_bot
BrowserMatchNoCase AhrefsBot probably_a_bot
BrowserMatchNoCase ScoutJet  probably_a_bot
BrowserMatchNoCase Yandex  probably_a_bot
BrowserMatchNoCase moget  probably_a_bot
BrowserMatchNoCase ichiro  probably_a_bot
BrowserMatchNoCase Yeti  probably_a_bot
BrowserMatchNoCase RavenCrawler probably_a_bot

Order Deny,Allow
Deny from env=probably_a_bot

The above should block a few crawlers/bots, but is by no means exhaustive.

Remember, that only good/honest bots are going to tell you that they are a bot in their user-agent string. Malicious bots are going to randomly selected real user-agents or tell you they are IE6 or something. So, the above is only to help prevent the site from being indexed, but not accessed.

It is also been reported that some search engines use a non-bot user-agent to get mobile views, so this should not be 100% relied upon to keep your site from being indexed and you should periodically check to make sure it hasn’t been indexed by any of the big search engines.

If you want to get even more aggressive or if it is important that the site remains private while you are working on it, you can block everyone and only allow access to select IP addresses using the htaccess file.

Another considerations is that, especially on a live website, you will almost never be able to just do an export/import(like we did in step number 3) when you are done working on the development website and want to transfer your changes over to the live website. You can easily copy the template/design/plugin files back over, but since your live site may have had orders, visitors, searches, customers, products added/removed, and other changes to the database, you can’t safely copy it back over without using live data. So, you may have to manually do some imports, create an update script, re-do some config settings, or otherwise manually merge in your database changes from the development site back to the live website.

Back to Table of Contents

If you have any questions or notice any issues, please don’t hesitate to leave a comment and I will reply as soon as possible!