Blog

DrupalEasy: Getting ready to run your first migration

AI generated image of a Drupal monster ingesting books.

Migrating content into Drupal is an extremely useful skill for most Drupal developers. Often, the most difficult step in learning to do migrations is the first one – that of getting everything set up and running your first migration to ensure that everything is working as expected.

Most Drupal migration-related blog posts and documentation that I’ve seen either completely ignore the setup process or gloss over it before diving into the details of writing migrations. I like to ensure, both here, and when working with participants in our Professional Module Development course, that we ensure a solid understanding of this process to build not only skills, but confidence.

This blog post will explain how to set up and run your first (very simple) Drupal migration. The process I will outline is actually the same steps I do when beginning to write a custom migration – just to make sure I have everything “hooked up” properly before I start getting into the more complex aspects of the project.

I generally write migrations on my local machine, using DDEV as the local development environment. 

Configuration migrations vs. Plugin migrations

When creating a custom migration, one of the initial decisions to be made is whether you’ll write the migrations as plugins or configuration entities. I’ve always used configuration entities, but there are pros and cons to both approaches. Here, we will focus on configuration entity migrations.

There are some minor differences in the workflow presented below when using plugin migrations. For more information on the differences, I recommend Mauricio Dinarte’s article. 

Core and contrib modules used

If you’re planning on following along with this article, the following modules should be installed and enabled:

Drupal core: 

  • Migrate
  • Migrate Drupal

Contrib: 

Source database

This blog post will demonstrate importing a small portion of user data from a Drupal 7 site into a Drupal 10 site. Normally, the first step is setting up the source data: in this case a Drupal 7 database. I normally create a new d7 database on the same MariaDB server as the Drupal 10 site (using DDEV, this is quite easy) and then import the Drupal 7 database into it.

Next, we have to tell the Drupal 10 site about the d7 source database. This can be done by adding the following database connection array to the bottom of your settings.local.php file (which I know you’re using!):

$databases['migrate']['default'] = array(
  'driver' => 'mysql',
  'database' => 'd7',
  'username' => 'db',
  'password' => 'db',
  'host' => 'db',
  'port' => 3306,
  'prefix' => '',
);

Note the database key is migrate and the database name is d7. Everything else is identical to the regular Drupal 10 database credentials in DDEV. If you’re using Lando or another local development environment, then your database connection array may be different. 

Custom migration module

Next, we need a custom module to house our migration. The easiest way to create one is via Drush’s generate command:

$ drush generate module
 Welcome to module generator!
––––––––––––––––––––––––––––––
 Module name:
 ➤ My d7 migration
 Module machine name [my_d7_migration]:
 ➤ 
 Module description:
 ➤ Migrations from Drupal 7 site.
 Package [Custom]:
 ➤           
 Dependencies (comma separated):
 ➤ migrate, migrate_drupal, migrate_plus
 Would you like to create module file? [No]:
 ➤ 
 Would you like to create install file? [No]:
 ➤ 
 Would you like to create README.md file? [No]:
 ➤ 
 The following directories and files have been created or updated:
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
 • /var/www/html/web/modules/custom/my_d7_migration/my_d7_migration.info.yml

Once created, go ahead and enable the module as well:

$ drush en my_d7_migration

Migration group configuration

As this blog post will be writing migrations as configuration entities, create a new config/install directory in your new module. 

We’ll define a migration group to which all of our migrations will belong – this is what connects each migration to the source d7 database. Create a new migrate_plus.migration_group.my_group.yml file in the new config/install directory with the following contents:

id: my_d7_migration
label: My Drupal 7 migration.
description: Content from Drupal 7 site.
source_type: Source Drupal 7 site
shared_configuration:
  source:
    key: migrate
  migration_tags:
    - Drupal 7
    - my_d7_migration

The most important bit of this configuration is the source key of migrate; this links up this migration group with the d7 database, with the migrate key we previously configured. Then, any migration created that is in this group automatically has access to the source d7 database.

Simple sample migration configuration

Next, let’s look at a super simple example migration for user entities. This will not be a complete user migration, but rather just enough to migrate something. Many migration blog posts, documentation pages and other sources provide guidance and examples for writing migration configuration files (remember, this blog post is focused on all of the configuration and mechanics that normally aren’t covered in other places.) 

Here’s the user migration configuration we’ll use. Create a new /config/install/migrate_plus.migration.user.yml file in your custom module with the following contents:

id: user
label: Migrate D7 users.
migration_group: my_d7_migration
source:
  plugin: d7_user
process:
  name: name
  pass: pass
  mail: mail
  created: created
  access: access
  login: login
  status: status
  timezone: timezone
destination:
  plugin: entity:user

Reputable Drupal migration articles and documentation will explain that migrations need a source (extraction,) some processing (transform,) and a destination (where to load the data.) Often these three concepts will be called ETL. Each of these concepts are easy to spot in our sample configuration file:

  • The data is coming from the d7_user plugin (provided by Drupal core with the data source being that of the my_d7_migration group which we configured in a previous step.) 
  • The processing of the data in this simple migration is just field mapping (to: from.) Many migration configurations have transformations as part of this section, often provided by Drupal process plugins. 
  • The destination is Drupal user entities. 

Running the migration

Again, these instructions are specific to Drupal migration configurations when created as configuration entities. Instructions for migration configuration written as plugins are slightly different (and not covered in this article.) 

As we are dealing with configuration entities, they must be imported into the active configuration store – which, by default, is the Drupal database. This is easily accomplished with the drush cim –partial command. This must be run each and every time a migration configuration file is modified. This is one of the (few, in my opinion) downsides of writing migrations as configurations. 

Next, I often check the status of migration via the drush migrate:status command. When using the Migrate Drupal module, it is recommended to always use the –group option otherwise the output of migrate:status can get a bit messy (due to all the default migrations that will be displayed.) 

The drush migrate:import and migrate:rollback commands should be self-explanatory. Each can be used with either the –group option or with a migration name (as shown below.) I almost always use the –update option on migrate:import for updating previously imported data. 

Finally, keep the drush migrate:reset command in your back pocket when writing custom migrations. If the migration crashes, you’ll need to use this to reset its status from processing to idle in order to run it again. 

Here’s a full set of the command specific to the sample user migration, migration group, and custom module created in this article:

$ drush cim --partial --source=modules/custom/my_d7_migration/config/install
$ drush migrate:status --group=whatever
$ drush migrate:import user --update
$ drush migrate:reset user
$ drush migrate:rollback user

What’s next?

What I’ve presented in this article is 90% of what I regularly use when running migrations. Sure, there are a few edge cases where oddities occur, but I believe this is a solid base of knowledge to start with. 

Once all this makes sense, I encourage you to utilize other blog posts and documentation to extend the user migration we started, or write new ones based on other resources. Once you’re comfortable writing additional migrations, learning how to create your own process plugins is the natural next step. As process plugins are written as PHP classes, some knowledge of module development is necessary. 

Interested in learning more about Drupal module development? Check out DrupalEasy’s 15-week Professional Module Development course! 

Additional resources

Lead image generated by OpenAI.