The Associated Press

Working with our partners at The Associated Press to rebuild and migrate a critical platform.

Snapshot

The Digital News Experience platform is used to distribute ready-to-publish web content packages to member publishers all over the world. It is a source of advertising revenue for AP and its publishing customers.

AP had quickly outgrown the initial launch of their Drupal-based platform. It couldn’t scale, was not architected for agility, and didn’t meet customer expectations. AP hired DevSpark to help re-architect, rebuild and migrate the platform to improve efficiency, reduce costs and satisfy a wide range of customers eager for additional functionality and greater ease-of-use. Through a close collaboration between the AP and DevSpark engineers, the Digital News Experience has been transformed to reduce back-end complexity and meet stakeholder needs. It is now the fastest growing product by revenue at AP.

Business

Challenges

With the Digital News Experience already being used by 300 publishers and generating revenue, AP sought to accelerate the growth of the platform, which would require greater scalability and new functionality.

As the contract with their Drupal managed services provider was nearing its end, AP wanted to work quickly to migrate to AWS so they could avoid bearing additional costs and have more control over the platform. To that end, it was critical to scale-up an experienced team and initiate a close working relationship quickly.

Results

The new Dashboard has helped the Digital News Experiences continue to grow and become AP’s fastest-growing product by revenue percentage.

Technology

Challenges

Time
Given current-state inefficiencies and because customers were eager for new functionality, AP needed to work quickly to address critical issues and then build a team to develop a long-term sustainable solution to meet the business need and keep revenue-generating stakeholders happy.

Web Services Migration
The platform was being served through Acquia, which did not offer AP the scalability and performance they needed for the Digital News Experience. The move to Amazon Web Services was mission-critical, but brought new challenges of large-scale migration, developing a new architecture and rebuilding many API components.

Drupal, and More
The existing platform had been constructed on Drupal, and AP needed a team of experienced Drupal developers and architects. The team also required developers experienced in PHP, Amazon Web Services and API configurations.

Many Sources & Complexity
The Digital News Experience needs to ingest content from many sources and in many formats and distribute content to publishers across the US. For AP, it was critical that the new platform be flexible and scalable enough to handle the complex demands of customers and stakeholders.

Big & Complicated Database
With a huge amount of content and a rich tagging system, the size and taxonomy of the database became hard and expensive to manage. The new solution had to reduce the size and complexity of the database without sacrificing their customers’ ability to discover relevant content.

Harvesting Institutional Knowledge & Collaboration
The engineering team at AP had tremendous knowledge that would be invaluable to developing a new solution. AP needed a firm that would become part of their team and work closely with their engineers not as a vendor per se, but as an extension of AP’s team.

Results

66% reduction in database size and a 72% reduction in taxonomy rows.
Taxonomy restructure cleanup to improve performance, including de-normalization of relational databases, implementation of caching capabilities, addition of configuration-based pagination capabilities, filtering, taxonomy index sharding, implementation of MySQL stored procedures and a comprehensive analysis of the Drupal database abstraction layer to fine tune configuration to optimal levels.
Led a successful large-scale migration of the platform from Acquia to AWS Architecture, implementing services such as CloudSearch, S3, EC2, Varnish cache, ElastiCache, SWF, RDS, SQS.
Successfully implemented javascript-based frameworks to improve development of Drupal themes using Zurb Foundation.
Led successful content ingestion, implementing AWS Simple Workflow (SWF) to send pre-processed content via RESTful API to a custom Drupal module, to add articles, photos and custom content.
Successfully implemented a RESTful API using a Drupal Services module to connect different Drupal websites.
Implemented reusable widgets that surface third-party content fed via RSS, to enable a consistent look and feel.

“

“The platform had accumulated significant technical debt, the code wasn’t documented, there was a lack of architectural leadership and we lacked in-house Drupal expertise. We needed to quickly find a trustworthy partner who would take ownership and responsibility for the platform as if it were their own. That’s what we found with Spark Digital.”

-- Vincent Tripodi, VP Engineering - The Associated Press.

“Spark Digital has enabled us to continue to grow and has helped ensure our products meet the unique needs of our customers in what is an ever-changing landscape, all while providing us with a platform that has room for further growth.”

--Kyle Magee, Senior Product Manager -- The Associated Press

“Revenue, new customer acquisition, and customer engagement all improved dramatically as a result of the new dashboard, while simultaneously reducing the amount of technical effort required to operate the product.”

--Kyle Magee, Senior Product Manager -- The Associated Press

“All the words you would use to describe a good platform apply: extendable, scalable, stable and resilient. The end-state was well understood, which made for an easy hand-off at the end of the project.”

--Vincent Tripodi, VP Engineering - The Associated Press.

“The AP has worked with Spark Digital on a number of additional projects since the launch of the dashboard. The team works hand-in-hand with our own staff to build a product that exceeds our expectations time and time again, flexibly shifting course to meet our needs whenever necessary. That flexibility and positive attitude toward our projects has made our daily communication and overall experience with the team very enjoyable.”

-- Kyle Magee, Senior Product Manager - The Associated Press

“Spark Digital bent over backwards to align around budget and velocity. We would not have been as successful without Spark Digital's responsiveness.”

--Vincent Tripodi, VP Engineering - The Associated Press.

“It felt like there was no distance between us.”

--Martin Meliendrez, Development Manager, DevSpark

”

Our Approach

First, we worked with our partners at AP to define an SOW that prioritized the work. We made sure critical problems were addressed first. Of course, this required that we quickly assemble a team of engineers experienced in the core technologies, including Drupal 6 and 7, PHP, MySQL and Amazon Web Service technologies including Amazon Cloud Search and AWS Simple Workflow (SWF).

At the same time as we assembled talent, we began a rigorous review of the current-state product and worked with AP engineers and DevOps managers to help establish an architectural vision and roadmap, and develop a product backlog.

The Team

Within two weeks we had a team of eight engineers working to tackle critical issues. After early success, we expanded the team to 12, including a Technical Lead who worked with AP’s Lead Architect to make recommendations that would support overall platform efficiencies and support a smooth migration to AWS.

Working with the Client

Using an agile methodology and working closely with AP’s team we quickly got into a working rhythm with weekly standups and demos every two weeks. Throughout the ongoing work we worked as an extension of AP’s team, touching base daily and tracking progress using Trello and Yodiz.

Throughout the entire project we made sure to communicate every relevant detail, success and roadblocks alike. By building relationships engineer-to-engineer and removing gatekeepers we produced results quickly and reduced stress for everyone.

Case Studies