Incremental Improvements: We’ve Just Eliminated 441 Days of Waiting for Listings to Appear

By Tim Dain

The Backstory

How we got here is decades in the making. For many years we relied on outside vendors to maintain our listing input and maintenance. Then in 2018 we created our own Add/Edit software and database and became co-owner of the data distribution technology (MLSGrid).

This essentially provided control of data input, data stored and data output. The primary reason for this change was to protect your most valuable real estate asset—your listing data. It ensured that listing data doesn’t become “trapped” within a vendor system that is difficult to access, convert or leverage.

Creating the Right User Experience

In the five years since we’ve launched the NorthstarMLS Add/Edit, one of our challenges has been related to the technical hurdles associated with breaking apart the monolithic, front-end systems you think of as your MLS.

These front-end systems—FlexMLS, Matrix and Paragon—can do everything from listing input, storage, display, output for you, all-in-one. Since they are self-contained, they have speed advantages that are hard to duplicate when using separate systems with different technology owners for each function.

Increasing Speed by 10,583 Hours

Reducing the time it takes a listing to appear in the MLS front-end systems after adding it in the NorthstarMLS Add/Edit system is one of our top technical challenges.

Where the slow-down happens is a data integration process called Extract, Transform and Load (ETL). ETL combines data from multiple sources into a large, central repository using business rules to clean and organize the raw data and prepare it for storage, data analytics, machine learning, etc.

Speeding this process has been a top priority, and we are currently upgrading the technology used to extract the listing data to offer improved performance, enhanced efficiency and better scalability.

I’m pleased that our focus and effort are starting to pay off with tangible results for the majority of NorthstarMLS subscribers. In the last week of May we changed the Matrix ETL process so that it would run every 5 minutes versus every 10 minutes. This one change reduced the time from saving a listing in Add/Edit to seeing the listing in Matrix by half.

It might not seem like a lot, but with over 130,000 listings per year flowing through NorthstarMLS Add/Edit, that’s an accumulation of 10,583 hours (just under 441 days) of waiting for listings to appear in Matrix that we just eliminated on an annual basis!

Yet, other opportunities exist to reduce load time. After running the ETL process to fetch listing data it takes Matrix anywhere from 2-5 minutes to complete the transform and load portions, so they are visible, searchable listings. We’re now working with Matrix vendor CoreLogic to reduce the timing of that portion of the ETL process. They have already realized an additional 40-50 second reduction through some adjustments in the process.

We’ll spend a couple more weeks finding incremental improvements that should add to the overall speed of moving listings from the NorthstarMLS Add/Edit system to the Matrix system.

What’s Next

Next on the horizon is the launch of our new Web API, which should provide additional improvements to the ETL process. The application programming interface (API) provides the ability to move data from one system to another much faster. It has been loaded onto our production servers and several vendors have started testing, including Flex, Matrix and Paragon.

We’re also going through the Real Estate Standards Organization (RESO) API certification process. Once the API is adopted by Matrix and in use with the ETL, our goal is to have this total process take considerably less time for listings entered into Add/Edit to appear on subscribers’ front end of choice.

I’ll continue to provide updates along the way. Meanwhile, we hope you feel the speed this selling season!

Tim Dain is CEO of NorthstarMLS. An industry innovator, Dain has served as vice president and general manager for real estate technology company Remine, as President and CEO of MARIS MLS, as Director of MLS for ACTRIS and SIRMLS, and as a REALTOR salesperson and broker-owner. He was named to the Swanepoel Power 200 list of the most powerful and influential executives in the residential real estate industry in 2017 and 2022.