Deepening code reusability with LookML project import
Mar 28, 2019
Each month, Looker releases new updates and features that further enable the smarter use of data. As we continue to improve and build upon Looker, we want to highlight and share notable features so that our customers can take full advantage of them.
With Looker 6.8 come many great additions focused on modeling. Most notably, we are releasing beta support for importing projects from private, remote LookML repositories.
To understand why this is a big deal, let’s first review why LookML is so valuable to data modeling.
D.R.Y. data modeling with LookML
One of the core benefits of LookML is that it stops you from repeating yourself when doing data analysis.
When writing SQL, you often look back at queries you’ve written in the past, copying and pasting little snippets of those queries, and reassembling them to form a new query. This process is error-prone and doesn’t scale well. With LookML, you write down those snippets just once into universal and reusable query components that are easier to manage. In computer science, this concept is called D.R.Y., or Don’t Repeat Yourself, and it can save you a lot of time (and tears).
Once the model is written, Looker is able to read it and present it in a visual, point-and-click interface that’s accessible to non-coders. As the user builds their query, Looker does the tiresome work of assembling optimized SQL code automatically behind the scenes.
The mantra is to write the code ahead of time, not every time.
The issue: sharing code between projects
LookML models are organized as projects. Projects are the highest organizational-level containers for LookML, and until recently it was not possible to share code between them. This inability to easily share code between projects can create issues in common use cases where different departments would benefit from being able to share the same code.
For example, say the finance department has a project that describes and calculates net revenue. The marketing department might also want to leverage this measure to determine the profitability of marketing campaigns. If they happen to be using different LookML projects, the only way to reuse the code would be to copy and paste it, which would take time and be subject to errors - not D.R.Y.!
Fortunately, problems like these have already been solved in the greater world of programming languages. The solution is called package management, and it is a feature of almost all mature programming languages.
A package is a unit of code (or software) that can be reused by another program. The package manager is the system that allows packages to be found, installed, configured, updated, and removed. With a package manager, the developer doesn’t have to do these tasks manually, thereby eliminating work and improving code consistency.
To apply these principles to LookML, we built project import. Project import allows the developer to reuse code from another project without having to copy and paste it, thus deepening our commitment to D.R.Y. coding and saving LookML developers valuable time. We first introduced project import as a beta over a year ago, and have since made many iterative improvements
Project import for LookML Blocks
The first, most obvious benefit of project import is in using Looker Blocks. Looker Blocks® are reusable units of LookML code based on common data sets and use cases. For example, we have a Salesforce Block that quickly adapts the notoriously tricky Salesforce schema and models it into dimensions and measures that everyone can understand. For anyone who has tried analyzing raw Salesforce data or has attempted to use its API, this is pure magic.
Project import improves the Blocks experience by allowing the developer to simply reference the URL of the Block’s source code and then immediately use that code inside their project. Additionally, by not hard-coding the Block’s contents, developers can opt to update the installation of the Block very easily, so as the Block continues to get better, so does the developer’s model.
Project import for the enterprise: the “hub and spokes” architecture
In large enterprises, the deployment of Looker can grow to a size where the core data team can no longer manage every department’s data needs, and some level of delegation must happen. Splitting up projects so that each department can manage their own becomes advantageous, but how do you maintain consistency?
In this case, you can use project import to push centralized, highly governed business logic down into each department’s project. We call this the “hub and spokes” architecture.
In the hub and spokes architecture, a central data team manages a core project (the “hub”), while each department manages its own projects (the “spokes”). The hub contains universal business logic, whereas the spokes contain domain-specific logic. With the latest release of Looker 6.8, hub logic is imported into the spokes via project import.
With the hub and spokes architecture and project imports, you get the best of both worlds. Hub and spokes allows everyone to run at their own pace with distributed, scalable modeling ownership, while project import ensures everyone is still speaking the same language of shared logic, derived from organization experts.
Ready to learn more and get started with project import? Check out our documentation, join the conversation about Looker 6.8 on the Community, and subscribe to our blog for future updates highlighting features coming with new releases.