Whether it’s migrating to a new storage system, database, application or entirely new server, data migration is a frequent project on business agendas. It is also a somewhat difficult and sometimes-dreaded process that requires business leaders to consider budget, resources, security, timelines and a variety of other factors for success.
SEE: Checklist: PC and Mac migrations (TechRepublic Premium)
As migration projects become more commonplace and complex, learn what trends are driving the data migration market and individual projects today.
Top 5 data migration trends
1. Shifting toward data lakehouses
Arguably one of the biggest innovations in data migration has been the data lakehouse, first introduced by Databricks, which largely does away with the need to migrate data. Rather than moving data from a data lake to a data warehouse with expensive and time-consuming extract, transform and load tools, the data lakehouse basically transforms a data lake into a data warehouse.
As Databricks executive, David Meyer, explained in an interview: “Data lakes were great in a lot of ways … but they didn’t have a lot of characteristics that you’d want to do data and AI at scale.” He went on to describe some of the weaknesses of data lakes, including their lack of governance, ACID compliance and transactional features.
SEE: Data governance checklist for your organization (TechRepublic Premium)
By adding a layer like the open source Delta Lake that Databricks uses, companies can leverage massive quantities of data for things like machine learning applications without necessarily having to move or migrate that data.
2. Avoiding data loss and expanding capacity through cloud migration
This second trend is far more obvious because, quite literally, almost everybody’s doing it. That is migrating data to the cloud. Though cloud spending remains relatively small compared to the overall IT market — less than 10%, according to IDC and Gartner data — it’s growing much faster than other areas.
There are a number of reasons for this. First, storing data within on-premises environments can lead to data loss. Even with backup policies in place, data storage is significantly more prone to failure on-premises than in a fully managed cloud environment where backups are automated.
SEE: Cloud Data storage policy (TechRepublic Premium)
By moving data to the cloud, enterprises can not only handle a wider array of data types, but they can also ingest more data much faster. They also get the benefit of seemingly infinite capacity, something they definitely don’t have with on-premises deployments, where the surge in data is maxing out companies’ ability to store it all.
Fortunately, each of the major cloud providers offers a variety of services to enable more seamless data migration. There are also a host of system integrators with expertise in assisting companies as they move their data to different cloud providers’ storage, database and related systems. It’s never been easier to migrate data to the cloud.
3. Uniting legacy on-premises data with cloud customer data
Part of the drive to migrate data to the cloud is that so much new data already lives there, and that data is in fact born in the cloud. This isn’t so much a trend in data migration as it is the reason for data migration.
SEE: 4 steps to purging big data from unstructured data lakes (TechRepublic)
Indeed, much of a company’s most valuable data, at least as it relates to customers, is cloud data, which has sparked data migration projects to move more hitherto on-premises data to those same cloud environments. This includes migrating data lakes and data lakehouses to the cloud. Through this process, organizations develop a richer, more holistic view of their customer data.
4. Using data migration resources to get the most out of unstructured data
An increasing percentage of the customer-related data mentioned in Trend #3 is semi-structured or unstructured data. Examples of these types of data include geospatial, sensor and social media data.
SEE: 5 tips to improve data quality for unstructured data (TechRepublic)
This data doesn’t easily fit within a relational database, and increasingly finds a home in so-called NoSQL databases. Whether unstructured data is stored in a NoSQL database, a data lake or elsewhere, enterprises are looking to data migration strategies and tools to move, cleanse and transform this data to make it easier to analyze.
5. Migration begets modernization
When companies begin to think about data migration, application modernization often isn’t far behind. If a company is considering moving data to the cloud, rather than starting with a basic lift-and-shift approach, many companies are now starting the process by moving from a self-managed, on-premises database to a fully managed database service. For example, these organizations may move from self-hosting MySQL to running Amazon RDS for MySQL.
SEE: What is data modernization? (TechRepublic)
The pre-migration stage is also when enterprises are increasingly choosing to rearchitect an application to use an entirely different database. Perhaps they are switching from a relational database to a document or key-value store, or perhaps they are moving in the opposite direction if they are starting with a wide-column database for a particular application and think a relational approach would be a better fit.
The minute you start to think about migrating data, it’s worth considering if it’s time to make other major changes to your data storage, management and infrastructure. You may also want to consider hiring data professionals who specialize in this kind of data modernization and migration work.
Disclosure: I work for MongoDB but the views expressed herein are mine.
Read next: Top cloud and application migration tools (TechRepublic)
