BI – Abylon

Effective Master Data Management with the Abylon Rapid Platform

Zoltán Zicherman — Mon, 16 Jan 2023 14:51:27 +0000

Effective Master Data Management with the Abylon Rapid Platform

Who needs Master Data Management?

If You already know, that you need Master Data management, feel free to skip this part and jump right here! This quick overview helps you to understand the business needs why a company might need a master data management tool and to provide some real-life business use-cases.

Imagine a Business Intelligence (BI)-system! – Ok, but how? – You might ask, and really, what is the substance in business terms of a BI-system (BI as Business Intelligence). I won’t be going into that in academic terms, but one way to look at BI-systems is that:

„A Business Intelligence system is a solution package, which gathers data from one or several source systems in order to create reports from it.”

Please also let me hereby express my apologies to true BI-developers. They probably have a much better definition, but for now, and for the sake of simplicity, let’s stick with this definition.

Imagine, that you have a BI-system. It works with data coming from various source systems, data like: users, customers, vendors, and so on.

Use case I. - Basic categorization

You want to categorize your vendors in a new way, to make a new report but in your ERP system (a source system for the BI in this case) there is no field or option currently for that kind of categorization, and it would take a lot of resources to add that field. This custom development is not on the development road map, but you still have to create a report using this categorization. Maybe you have to build reports on this ground, maybe you have to build/develop even more operational stuff.

Use case II. - Tax report change

Imagine that for some reason the international tax-environment changes (or will likely to change very soon), and the reports you’re taking care of need a rapid readjustment for planning, but operationally you are not able to change the source-system’s tax-categorization data.

Use case III. - Integration of new ERP system

Your company is implementing a new ERP-system, but you have one or multiple legacy systems, which have dimension attributes (for example: categorizations of vendors) in the legacy system(s) and in the current bi-system, which are already in place, but the new system is based on a different logic, thus this kind of categorization will be replaced, but you don’t want to change all your reporting structures straight up since the new system will be implemented in the long-run, country-by-country.

Use case IV. - Affiliates with different accounting systems

You are working with several affiliates who use various, custom accounting source systems and different book of accounts structures. However, you still need to create reports using their data. This means you need to manually consolidate all of this data and arrange it into your own data categories, dimensions and reporting structure.

Downsides of the "old-school" approach to Master Data Management

If you’re coming from business (as I do) your first idea to solve any of the above use cases is to gather all the data in Excel-files. Then, eventually put them in a some SharePoint folder and then share the Excel files across multiple users throughout the globe. Right? This is a legit approach right?

Yes it is, if you’re absolutely comfortable with the „Demons of the Shadow IT” haunting you utill the end of your practice at that company… I know what I’m talking about. I’ve built the shadow IT for years… Here are dome of the problems with this approach:

To put it simply, who should be able to access the data, and who should be able to adjust it?

How to manage if more than one user wants to work in it simultaneously? I know-I know: Excel is supposed to handle this. However we know there are issues there, like not saving something, if two or more persons are working on the same file at the same time.

How to know, what has been adjusted, who made the adjustments or edited the data (God-forgive deleted or added data), and what was the value before those actions took place? Actually this is a very common issue, when more than one person works with data. How can one be sure, that nobody touches their data, or if someone touches it can be traced back, so maybe there will be a discussion about the causes of the matter.

How to validate the values? Okay, Excel’s data validation is a viable an option here. We use it too. The problem is, that it can be easily overwritten, so it’s not a bullet-proof solution.

What if there are more complex validation rules that have to be applied? Validation based on some sort of business logic.

Imagine that you can only add data in a column which is between the min and max values of another table’s column’s data. Like you have to set monitor sizes, and there is a possible minimum and maximum value stored somewhere based on the available stock. How do you handle this?

Business Intelligence with adequate Master Data Management

Let’s start with usability! Just like with any other IT solution, usability, and therefore the User Interface is a key factor. What’s the best UI in this case? If you are guessing I hope you got it right! It’s the one and only MS Excel. I’m not joking. Excel is still the most universal (and most popular) tool for every controller or even office worker dealing with data.

Of course it has it weaknesses, so we gave our usual Excel UI some superpowers! We connected it to a database through our Excel add-in available directly in Excel itself. So what are the benefits of that? Let’s see a few:

Since we are connected to a standard SQL database, this connection can be used for any other standard purpose like being a data source for your queries, reports, or BI-system built somewhere else.

The back-end of the solution is a standard SQL database so most changes or developments to the solution only require SQL knowledge. Considering most modern enterprises usually have their own architecture and database-administrator team and often a few BI professionals as well, it means that the customer itself can participate in the development or customization of their own solution.

Once we are familiar with the solution, it is an easy task to configure and create new tables, or adjust existing ones.

Since the user interface is Excel itself, there is no need for business users to do any type of programming, they can simply take care of the data in Excel.

The solution has a standard way of keeping track of history data, and logging events, including information on who viewed the data and when.

You need to validate data based on columns in other data tables? No problem! All you need is a simple configuration setting to reference that column. Once configured, data will be validated inside Excel using its own validations, plus it will also be validated in the background. Additionally – and more importantly – the data will also be validated in the “background” (on the back-end, database level)

Simple validation rules are not enough and you need to have custom validations based on business rules? Again, no problem! Any rule that can be described logically (and is a straightforward logic), can be developed in SQL and placed in a custom validation script, so that your data can only have values which are allowed!

Modern enterprise-grade MDM in just a couple of weeks

I’m not a marketing person. I don’t like big words. I used to be one of those people who are ultimately the end-users of these solutions. Based on that experience I can say that once your company starts doing Master Data Management with the Abylon Rapid Platform is won’t just be a small step, but a huge jump toward efficiency. You will have a automated, flexible, easily customizable solution that radically reduces all the manual work related to any MDM-related task.

Moreover, because our solution is built on top of Excel – a familiar and widely used tool for users involved in these tasks – the implementation won’t take more that a couple of weeks!

If you want to improve your Master Data Management and what you just read sounds interesting give us 30 minutes of your time to present you our solution and show you that all the above is not just talk!

Found this post interesting?

Subscribe to our newsletter to receive updates of similar post and news!

Follow us for more news and technical insights!

Five Biggest Challenges in Financial Planning

Master data custom validation using the Rapid Platform

Unleashing the Power of Excel: From VBA to Power Query

UK market here we go – Abylon’s International Ascent

The post Effective Master Data Management with the Abylon Rapid Platform appeared first on Abylon.

3+1 reasons why you need an Advanced Analytics Platform

Zoltán Zicherman — Fri, 09 Sep 2022 16:26:23 +0000

What is Advanced Business Intelligence or Advanced Analytics?

By Gartner’s definition: “Advanced Analytics is the autonomous or semi-autonomous examination of data or content using sophisticated techniques and tools, typically beyond those of traditional business intelligence (BI), to discover deeper insights, make predictions, or generate recommendations. Advanced analytic techniques include those such as data/text mining, machine learning, pattern matching, forecasting, visualization, semantic analysis, sentiment analysis, network and cluster analysis, multivariate statistics, graph analysis, simulation, complex event processing, neural networks.” (Source: Gartner Glossary)

In practice Python coding combined with some sort of Machine Learning development is usually called Advanced Analytics. You can realize business value from its output, but this output should be stored somewhere, as well as its input. To get the most out of Advanced Analytics, everything should be designed, planned and connected, which is usually much easier to do within a Platform.

Why would anyone need an Advanced Analytics platform?

1. Business benefits

All companies handle data. More and more every day. The magnitude only depends on the size of the company. Data from sales transactions, internal master data, product categories, etc.

This data has huge business value. The proper value can be discovered by data professionals, who can tell the current figures from past/present sales data, and can also predict the future, count efficiency rates, net margin, and many aspects you are curious about. But we can agree that without a clear understanding of this data a company can’t be successful in 2022.

According to Forbes, 95% of businesses cite the need to manage unstructured data as a problem for their business.

2. Technological modernization

Data professionals agree that data is still best stored in a database. Data professionals also agree that there might be more opportunities in raw data stored in databases, but it is usually two different worlds, relational databases and machine learning are not really close to each other.

The bridge between them is an advanced analytics platform, that can understand and serve both worlds very well, bringing out the best synergies from each.

3. Sustainability: lower and more dynamic costs

Locally: The platform’s costs are dynamic (use less, pay less), but also accurately predictable (hourly rates). Also, you don’t need to hassle with technical physical parts, and there is no need to keep a team of technician handymen standing by until they must urgently fix a serious outage. There is no such outage in Azure.

Globally: Way less energy consumption, and it is only consumed when it is really used. There are no idle blank spots (day vs night energy consumption) and the energy bill is paid by Microsoft, NOT you.

+1 The future of BI is in the cloud (or multi-cloud, or hybrid)

Not so long ago, all big companies stored their business data in their own, huge on-premise data centers. But maintaining and operating this type of infrastructure costs a lot of money and with the size of data growing exponentially this concept seems like a dead end. This led to the birth of the concept of cloud computing, offering a new era for data-oriented systems. Cloud computing is a lot more than a catchy buzzword; it is the next generation infrastructure that modern companies already use.

If your company is not on board yet, you’re already losing out to your competition. In a 2021 O’Reilly survey, 48% of participants said they plan to move most of their apps to a cloud during the next year.

While according to Gartner, companies will deploy 95% of new digital workloads on cloud-native platforms by 2025.

There are options even for those companies who have their security and data confidentiality concerns with the cloud. Services like Microsoft Azure allow you to keep some data on public servers while giving you the option to keep sensitive data on private servers so that you can actively control and monitor it.

An Advanced Analytics Platform is naturally built using cloud technology, and as cloud computing is the future of data systems, an advanced analytics platform is the future of analytics, reporting, and self-service BI.

How to get started?

Well it always depends on the type and size of your company, on the industry operate in, on your business case and your cloud and business intelligence “maturity”.

Abylon Consulting has delivered more that 200 successful BI projects in the past few years including Advanced Analytics projects utilizing Microsoft Azure and Synapse technology, so if you’re looking for a someone to fast-forward your company to modern era of data analytics contact us and let’s have a quick talk to see how we can help you!

If you are interested in the technical side of an Advanced Analytics Platform and you would like to give it a try check out this post that describes how to create a complete framework around cool buzzwords like “Python, ML and AI” working with industry-leading Microsoft products.

The post 3+1 reasons why you need an Advanced Analytics Platform appeared first on Abylon.

Abylon Consulting’s experience with railway BI and data projects

Zoltán Zicherman — Tue, 19 Jul 2022 12:26:41 +0000

Railways - a world of diverse, complex and challenging BI and data projects

Abylon Consulting and the Hungarian Railways

Hungarian railways have been a priority sector in the life of Abylon Consulting practically since the company was founded. We have been working together successfully for 5 years now and we hope to continue our professional cooperation well into the future.

The planning, organisation and monitoring of rail transport is a very complex process. The individual areas involved are themselves highly complex units, whether it is maintenance, sales or traffic management.

It is often difficult for an IT service provider, consultant or developer to adapt to the specificity of the rail sector. Beyond the physical reality, there is also a complex legal regulatory environment to adapt to.

On the other hand, it is this complexity, these specific needs, that also present an interesting challenge for BI companies dealing with large amount of diverse data like Abylon.

At Abylon Consulting, we pride ourselves on our many years of cross-disciplinary experience in rail IT. The results of our BI projects serve real business needs, help strategic decision making, and also help track day-to-day business operations. Understanding needs, data and customer satisfaction are a top priority in our projects.

Rail BI and data project delivered by Abylon

Below we would like to give you a taste of some of the areas where we have been working together with the Hungarian railways in recent years:

Passenger sales
Train traffic, events affecting train traffic
Passenger count/occupancy data
Infrastructure sales
Vehicle technical condition monitoring
Measurement of staff performance

These projects involved a total of 13 data models/data marketplaces operating in an on-premise environment among the Hungarian rail transport stakeholders, as well as a cloud pilot project.

Our solutions work in a Microsoft environment, using SQL Server, SSRS, SSAS, SSIS, C# and of course Excel, which is of paramount importance for the users. Projects are well structured, following the structure required by Abylon Rapid Analytics and our best practices.

As data markets continue to expand, we are actively assisting our customers with self-service reporting, which is an additional Power BI Desktop model.

The data markets and data warehouses are served by the Abylon Rapid Platform, with several special features designed to precisely meet client needs.

For most of our data models, we provide an additional value add in the form of our advanced map-based data visualization solution built on Power BI – Abylon MAP, which provides a map-based view of any metric.

In future posts, we plan to go into more detail about some of the real business cases and the related projects, of which we are justifiably proud. But for now, here are a few brief summaries of some of the projects we’ve worked on. Why they were interesting or challenging and what we’ve learned from them?

Near real-time data datawarehousing

Short description:

Usually data warehouses are updated daily. Practically, these updates are performed during the nightly idle periods so that the daily business is not disrupted by the load involved. However, in the case of traffic and related events, it was essential to update the data warehouses more frequently, several times a day, in order to be able to check the current, real-time data throughout the day.

What we learned:

It proved to be an exciting task to implement, resolve and optimize the intraday loads without any performance degradation on other systems, both in case of SQL and tabular models. We needed to tap into technologies that only enterprise application suites provide. Using our own framework, we had to choose a complex partitioning solution that handles current data in daily partitions and recent data in monthly partitions. Read more about this project here: Near real-time data warehouse

JÉSTAT+ Route decomposition and display of ticket dat

Short description:

A breakdown of the route description of the tickets sold according to the infrastructure included in the published timetable. Difficulties have been encountered due to the textual nature of the route description and the fact that some sections cannot be clearly assigned to railway lines (e.g. the PDF timetable available on the MÁV website suggests that some sections may belong to multiple lines, etc…)

What we learned:

With a thorough understanding of business logic and experience in the field, using SQL and .NET code, the appropriate logical decomposition was achieved. Creating a tabular model capable of handling data (>10 years of total sales data) was also a big challenge that required a lot of additional research. Read more on this project in this post: Railway data map visualization.

START-TÁR Railway vehicle condition evaluation

Short description:

Determination of availability, uptime and productivity based on the activities and conditions of each passenger vehicle (railcars and multiple units). For example, not only major breakdowns affect availability, but also if a vehicle has been in maintenance (activity) for 6 hours, its availability is 75%, and so on.

Experience:

On the SQL side of the project, it was an exciting task to compare vehicle activities and states, assigning a rating indicator to the vehicle based on the parallel, simultaneous states and activities, according to a defined priority. An other interesting part of the project was the handling of idling and defining the rating between two activities/states.

An award-worthy solution

We are planning to participate in the international Awards for Excellence in Railway Publication, organised by the Union of Railways (UIC) with a recent BI solution we developed together with MÁV (Hungarian State Railways) which examines the impact of delay events on the rail network.

The solution is showing, through a user-friendly report, that a primary delay event can lead to additional so-called secondary delays elsewhere on the rail network, and thus cause a degradation in service quality.

A special business logic for the correlation between delays has been developed, which efficiently and almost synchronously collects the delay statistics caused by disruptions.

An example showing how one delay is effecting other lines, trains in the network.

Successful rail BI cooperation

We are very pleased with the many years of successful cooperation with MÁV. We are proud of the many successful projects and of the experience and know-how we have gained in the course of these projects. As a result we can safely say, that:

Abylon Consulting has one of the largest expertise in the field of railway data management, data analytics and business intelligence (BI) not only in Hungary but also in Europe.

Found this post interesting?

Subscribe to our newsletter to receive updates of similar post and news!

Fore more interesting articles and news follow our social media channels!

Five Biggest Challenges in Financial Planning

Master data custom validation using the Rapid Platform

Unleashing the Power of Excel: From VBA to Power Query

UK market here we go – Abylon’s International Ascent

The post Abylon Consulting’s experience with railway BI and data projects appeared first on Abylon.

SSAS Tabular model management with the Abylon Rapid Platform

Zoltán Zicherman — Mon, 23 May 2022 15:25:22 +0000

Intro

In the context of a number of recent enhancements, we thought it is worthwhile to write up a brief summary of the Tabular model services available within the Abylon Rapid Platform and highlight how they support development, processing and logging.

Development

First of all, during development, tabular models can be deployed to any number of environments just like ETL processes, databases, reports (etc.). There is no need to rewrite the connections every time you move to a live environment, the platforms will do it for you after setting up the appropriate parameters. It also provides the possibility to publish user-level self-service models to a productive environment.

The developer’s life is made much easier by automating the creation of dynamic measure tables. Dynamic measure is useful for any business, but the creation and maintenance of the necessary tables and measures is time-consuming and sensitive to changes. Abylon Rapid Analytics takes this burden off the developer’s shoulders. It also makes it easy to produce measures based on selected measures such as LY, MTD, ALL, etc. based on calculation groups.

Processing

Our execution plan manager allows the usage of several OLAP processing steps in each execution plan item. Such as:

Full process, this is obvious
Dynamic partitioning, code-driven version: dynamic partitioning creates the partitions according to the configuration and processes the appropriate number. Main features: rolling window, incremental partitioning, incremental processing, fix interval.
Dynamic partitioning, data mart-driven: the data market developer has the possibility to create his own stored procedure, which passes the processing and creation command to the framework.

Logging

Metadata backup: main data of SSAS tabular models installed on environments, up to the query level of partitions.

SSAS instance monitoring: load, queries monitoring with Platform Planning functionality.

This is just a short teaser because this list could go on and on. We believe that, as with the whole Rapid Platform, in this case our strengths lies in the convenience of the ready-made solution, the cost-effective customization and the fast, customer-oriented development.

The available features do not set a mandatory direction, but provide opportunities, and the developments made during implementation help to fully meet the needs of our customers and provide support and good foundation for further BI development projects.

The post SSAS Tabular model management with the Abylon Rapid Platform appeared first on Abylon.

Case Study: Advanced Analytics on top of a legacy on-prem DWH

Zoltán Zicherman — Tue, 08 Feb 2022 10:50:09 +0000

Introduction

In this post we’re going to describe an advanced analytics project we recently delivered for a large international company. The main goal of the project was to improve the management of our client’s retail data (sales data coming from their shops) by adding an additional advanced analytics platform on top of the existing infrastructure.

Business Challenge

Prior to this project all retail data was only managed by a large, traditional, monolith on-prem SQL data warehouse. Data coming from this legacy data warehouse is aggregated and presented in pre-made group-level dashboards which are limited and highly dependent on report developers. Accessing more detailed data or data outside of these dashboards, or creating custom reports based on it would require a lot of effort.

This rigid infrastructure was not suitable for advanced analytics which requires dynamic data and an option for self-service business intelligence (BI). Furthermore, our client has other BI solutions that would also require this type of dynamic data.

Solution – Advanced Analytics Platform with Azure Synapse and Databricks

We built a new, cloud-based (Azure Synapse-based) data warehouse on top of the existing on-premise data warehouse and connected it to the existing Databricks (DBX) Platform. We extract the raw data from the on-prem data warehouse on a daily basis and make it accessible in our new Advanced Analytics Platform.

The new platform consists of two separate sub-platforms:

A Databricks (DBX) Platform for development and Business Value Realization solutions (for self-service development)
A Synapse Platform for transactional level analysis (for self-service analysis)

The Databricks Platform is a self-service development platform that uses Python, Machine Learning and Artificial Intelligence technology. It’s mainly meant for internal developers for forecasting and creating various data science solutions, and for external vendors (developers) who deliver custom solutions to cover their own business needs.

While the Synapse platform’s main goal is to enable self-service data analysis. Users can directly connect to the raw data from the data lake, execute custom queries, see the data on a transactional level and even connect to the data from Excel or Power BI.

More dynamic data

Compared to the aggregated dashboards (built on the on-prem data warehouse), in the new Synapse-based data warehouse all the data is available on transactional level. This means data like: cart size (how many items are bought together), all transactions, transactional data on item level (which items are bought together, discounts, sales quantity, etc) all meta-and master data (shop details, like opening hours, plot size, geo coordinates etc.) Most of this data was previously NOT available in the pre-made dashboards built on the legacy on-prem system, and now it is available in the Synapse environment, in a more dynamic set-up and to a much larger audience.

Easier access, larger audience

With this new advanced analytics platform, business analysts can also access all the data and create their own reports, even from Excel. This setup also opens the door to create an additional data layer for third party companies (like vendors who are selling their -typically beverage- products in these shops), which can even be sold as a service, to create an additional stream of revenue.

Scalability and reduced infrastructure costs

Another huge advantage is that the new platform allows our client to dynamically scale compute resources up or down depending on their needs. This means a significantly better solution in terms of cost effectiveness.

The post Case Study: Advanced Analytics on top of a legacy on-prem DWH appeared first on Abylon.

Advanced Analytics Platform – how to create a dream framework for Python/R

Zoltán Zicherman — Sun, 30 Jan 2022 17:16:24 +0000

Introduction

Advanced Data Analytics and Data Science is a really hot topic right now and it’s something definitely for the future as well. In 2020 the global data science platform market was valued at USD 31.05 billion, and it is expected to reach USD 230.80 billion by 2026, registering a CAGR of 39.7 % during the forecast period, 2021-2026. Impressive numbers, but how does it affect your life? Data is getting more and more important in nowadays world, and even you can play an important role in its expansion.

Since you clicked on this post it’s clear that you are interested in data analytics. Maybe you’re interested in the technical side (as a tech expert), or in the results of data analysis (as a manager) or you are just simply curios about this topic.

You would like to start working with some data, but your company doesn’t have the data infrastructure and the workflow for that yet. You’re not the only one who’s in this situation. As data science is going viral, more and more companies are realizing its benefits and thinking of investing in it on long term, but they don’t exactly know how to create a complete framework around cool buzzwords like “Python, ML and AI”.

In this article we describe how to create one. Since we have a lot of experience in working with industry-leading Microsoft products, in this example we are going to show you a setup that is based on the Microsoft cloud architecture, which is already tested, implemented and operated on a daily basis.

At Abylon Consulting we’re crazy about data, so if you like this topic browse through our blog for some other data analytics related posts and guides, or check out our product or service offerings if you are looking for a solution for your company.

What are my essential goals?

I want to work with Python/R/spark, since I know they are interesting and very fashionable nowadays. I mastered basic SUM, COUNT, AVG Excel functions long ago, I want something way more sophisticated.
I’m interested in more massive datasets: even public dataframes from the internet, even internal database tables or corporate files.
I want to visualize my results on good-looking reports, spiced with some automations.
I want to have a complete infrastructure that is robust and simple at the same time

To meet these expectations, we should get more familiar with the componets that deliver these features.

Introducing the components of our Advanced Analytics Platform

Good news, that all these needs may be solved with a few resources of Microsoft Azure:

Power BI (PBI)

Possibly you are already familiar with Microsoft’s industry-leading data visualization tool.
If you like charts, filters, plots, you will feel home. Call it a well-balanced mixture of Excel, Power Point and SQL.
Though it’s not placed in MS Azure, you will need it to create beautiful and informative reports.

Azure Databricks (DBX)

A Jupyter notebook-like interface which does the core data analysis and processing.
You can write scripts in Python, R, Scala, and even SQL. Supports Apache spark also.
It runs in the cloud, you don’t need to install any distibutions/libraries/etc on your computer.
Scalable compute resource, scalable cost (flexible and predictable, pay as you go)

Azure Datalake Storage (ADLS)

The cloud storage you will use. Simple, but very useful, consider it as a more strict OneDrive or Google Drive.

Azure Data Factory (ADF) – optional

If you need scheduled data copy from external/internal source systems, ADF will possibly be handy.
It is responsible for data movement activities in the cloud, or even on-prem.

High-level design

At the figure above you can see these components of the infrastructure. This basic design can help you to imagine the role and their consecutive manner. Let’s get through it in a nutshell:

First, you need to have some source data stored on the ADLS to begin with the analysis. Don’t worry, you can upload any flat files (like csv, json, xlsx) from your computer, there is no need for ADF at the moment.
You will consume & analyse this data with DBX, and store the output on the same ADLS.
If your analysis is done, and the result data is available on the ADLS, it’s time to open up a PBI file, and bind the necessary data from the source file(s) of the ADLS.
If you are finished with report development, you can publish your report to PBI Online, and share it with anyone.
If you have build a solution that is time-dependent, and needed to be refreshed from time-to-time, you can automate the execution of the DBX and the refresh of the PBI report in the cloud.
- If not only the calculations are time-dependent, but source data should also be refreshed, it’s time to add an ADF. With ADF, you can schedule data loads from source to sink (ADLS), plus DBX notebook executions can be triggered as well

Implementation

Azure tenant

To get all of that, you need to have az Azure tenant and an Azure subscription. You can try it for free for 12 months and you can use a credit of 200 USD in the first 30 days.

Resource Group

After the registration, create a new Resource Group (RG) which functions as a box to store your things. It doesn’t cost any money, it is only a simple “folder” in your subscription.

DBX

Create a new Azure Databricks (DBX) workspace, within this RG. Lauch the workspace, and take a look around there. You can find the menu options on the left, and maybe the two most important options are Workspace and Compute.

Workspace stands for a folder structure for your executable Jupyter notebooks (ipynb format), while
Compute holds all the compute resources (clusters), which execute your code in the notebooks

You will have to create a cluster prior to any development. You are charged by cluster time usage, so you are supposed to optimize the runtime and not leave any idle time. You can pick whichever setup fits you and your budget best.

Check this deeper guide with a case study if want to dive in DBX Python coding. Remember, core coding will be done here in DBX, all other components only help you with connectivity, data freshness, and robust operation. If you wish to try DBX alone at first, you can do it without creating any other resources. But, for a complete platform, you will need other components as well.

ADLS

Create a new Storage account (ADLS), also within the same RG. At Settings/Endpoint blade (left-side menu options are called “blade” in Azure) you can find the links to this specific ADLS, use this link when you aim to connect to this from PBI.

Connect ADLS with DBX

To make these two resources communicate with each other, it is needed to create a bridge between them. The best solution is an OAuth 2.0-based solution, named Service Principal (SP). This SP is an Azure Application, that creates the bridge between the two components, providing authentication from and to both parties.

The recommended option is a mount point, which is basically a link to a container of an ADLS, that the DBX notebooks can use as a link to point directly to the files placed on the ADLS.

Having a working SP provides a rather dynamic communication between the two parties allowing you to quickly access (write/read/modify) any files stored on the ADLS from DBX code. This allows you to create a flexible Machine Learning or Big Data solution, with serious inputs and outputs.

Power BI

From Power BI it is very easy to connect to ADLS, pick the ADLS Gen2 connector, sign in or provide your access key (preferably in a corporate network) available on Security+Networking /Access Keys blade of the ADLS.

Publish the report if you are finished, and set scheduled refresh in the online PBI service if it makes sense (based on your data).

ADF

If your source data changes or refreshes from time to time, you may need an ADF to solve the data processing. Create an ADF in the same RG you used before, set it up and get comfortable with it in a few steps.

You can create pipelines with activities, most likely a copy activity, of which you can set ADLS as sink, and your preferred data source as source. Both cloud and on-prem sources usually have authentication that you need to comply with the following way.

Create a Linked Service to store the connection information about the source system. It will basically stand for an independent data store. Every high-level information like id, connection string, username and password, other credentials are usually used here.
Create a Dataset within a Linked Service to specify the exact details of the source files/table you want to connect to (both source or sink). Typically foldername, filename, text encoding, schema name, tablename, other low-level details are used here.

Since ADF is in the cloud, for on-prem data loads you need to configure an Integration Runtime (IR) on the physical machine you are using. Either you are using recurring flat files from your home PC or refreshing database tables from the company’s on-prem SQL server, you will need to have a self-hosted IR installed on the physical machine to comply with the connection details.

Real-life example

If you are curious to see a real life implementation of a similarly built Advanced Analytics platform and the benefits it offers compared to traditional data warehousing check out this short case study: Advanced Analytics platform on top of a legacy, on-premise Data warehouse.

Some more tips & tricks, know-how and best practice!

Well done! As the architecture is complete, everything is ready to operate in production. All of these components have some more useful features to know about, briefly:

ADLS

- Security: Access Key cans be changed, reminders might be set
- Lifecycle mnagement: Data storage tiers may be set (hot-cool-arhive) that determine data access time and the mostly cost

DBX

- Cluster version: runtime setting is rather easy e.g. from version 10.0 to 11.0, just open the dropdown menu
- Install libraries: easy to install any Python/R/etc by PiPy, Maven, CRAN or from external link
- Access libraries on the cluster: Notebooks can only use the lib, if they are runing on the cluster on that the lib has been installed
- Data source: you can configure other source than ADLS only

ADLS-DBX SP

- Connections can expire: check at the SP (Azure App) when is it expiring, usually 1 year, but it can be set longer. To renew it, the whole process has to be done again (except Azure App creation).

PBI

- Data sources: it is smarter to use validation by access key, and use only one data source instead of many, if new files are from the same source, this case won’t mean extra work
- PBI service: to schedule a refresh, access keys must be added here also (again, smarter to use one source so that credentials are not to be added multiple times)

ADF

- Scheduled triggers: set periodical execution in the lowest detail one can imagine. Run pipeline on the 2nd and 3rd thursday of every odd month? No problem to set, very cool.
- Alerts: very useful to set alerts on pipeline run failures, it helps to get known about problems at the right time
- Debug vs Triggered run: triggered run can execute processes in paralel and has greater compute capacity than simple debug. As the name indcates, use debug only for debugging
- Git: use Azure DevOps for code management. Releases, publishes, merge, pull, can be done very well.
- Azure KeyVault: store your secrets in this Azure resource. Mostly ADF focus, but can be handy many other cases

Still curious for more?

If you feel that your appetite is just coming for more advanced analytics solutions, check:

Azure Synapse Analytics – if you want to quench your hunger with modern cloud database solutions and massive data workloads in the cloud, or take a bite to
Azure Machine Learning – if you feel like taking an adventure with the next generation data science solution of Microsoft (an extraordinary crossover of DBX, Rapid Miner and a REST API portal)

The post Advanced Analytics Platform – how to create a dream framework for Python/R appeared first on Abylon.

What clients say about Rapid Analytics?

Zoltán Zicherman — Tue, 16 Nov 2021 17:29:58 +0000

Interview: What our clients say about our DWH Automation tool

A few weeks ago we successfully completed a data warehouse project for Dashboard Solutions using our Data Warehouse Automation Solution – Abylon Rapid Analytics. In relation to this, we had a quick interview about with senior BI Architect Zoltán Nyircsák to get a quick feedback from the clients’ side.

Abylon: Thank you for taking the time for this quick interview. First of all, for what tasks do you use Abylon Rapid Analytics?

Zoltán Nyircsák: It is important to mention that we basically did not buy a technology or a component, but a solution. We chose Abylon and their data warehousing solution because the technology they used promised a quick rollout (and they kept that promise). But nonetheless, it was paramount that the experts were truly experts from both business and technology perspective. In fact, on the customer side we don’t even really need to understand what ARA (editor’s note: ARA stands for Abylon Rapid Analytics) does, it’s enough to articulate the business problem and then enjoy the benefits.

Abylon: How and to what extent does Abylon Rapid Analytics simplify your work?

Zoltán Nyircsák: Since the technological solution and its maintenance and development are not on the customer’s side, so it’s not easy to answer this question. However, we do enjoy the benefits from the business side. Eg: rapid development of data markets.

Abylon: How responsive is the solution (or Abylon?) How fast can it track changes?

Zoltán Nyircsák: The responsiveness is good. There were several source system changes that were easily handled by the solution. For example, there have been several overnight transitions to develop a new source system. Because we work with source systems that are in use, they are constantly changing, evolving, getting new features. So it was important for us that the BI solution we were building could handle this (with the help of Abylon experts, of course.)

Abylon: How flexible is the system?

Zoltán Nyircsák: So far, we have found a solution for everything together. Here again, I would like to point out that the technology used is “hidden” from us. This division of responsibilities between us and your experts was agreed prior to the project. Our part is to formulate the request, whether it’s some specific data, analysis, or business problem, and you give us the solution.

Abylon: How complicated do you find managing such a system?

Zoltán Nyircsák: This is also part of the service, so we don’t have to deal with it. However, the data warehouse and data market based on the solution, and the Power BI service, was an excellent choice. We are able to respond quickly to new business needs, facilitating the accurate, timely and reliable delivery of not only data but also information.

Abylon: What do you think, how much time did the use of Rapid Analytics saved you during your data warehouse building project?

Zoltán Nyircsák: Without it, it would’ve been an order of magnitude (I really mean the 10 times multiplier) more time and money to develop such a solution and then to adopt and react to all the changes. The development time has been largely reallocated and instead of spending time on manually creating the obligatory things, we were able to spend more time on the essential things and put more energy into the construction of the data market.

Abylon: Finally, what would be the feature that would make Abylon Rapid Analytics even more valuable to you?

Zoltán Nyircsák: Right now, we don’t really need anything else, but if I think about the future, maybe a built-in report on the operation of Abylon Rapid Analytics, the data loads and performance would be good.

If you are interested in our DWH Automation solution, head to the dedicated product page or contact us and we’ll be happy to answer your questions or arrange an online presentation

The post What clients say about Rapid Analytics? appeared first on Abylon.

How to post G/L records with retroactive data with the help of BI and SCD?

Zoltán Zicherman — Wed, 10 Nov 2021 15:04:57 +0000

For whom this blogpost is meant for

IT professionals who work with data, optionally familiar with BI or Datawarehouse technologies, and may need to create a solution in which they need to create posting files or an interface for accounting /ERP systems from historical data. This article can help them to identify the required actions and things to consider.

On the other hand, it is also intended for financial professionals or Business Analysts who are involved in a project or projects where the goal is to develop an interface between normal transaction systems and accounting/ERP systems.

Intro

What is posting in accounting?

If you know what an SCD, or even a SCD2 is, feel free to jump on to the task description. If you don’t know what they mean, the following few lines can help you understand.

As part of the month-end closing process, it is always necessary to make numerous manual postings in the financial accounting. Posting in accounting is when the balances in sub-ledgers and the general journal are shifted into the general ledger. Posting only transfers the total balance in a sub-ledger into the general ledger, not the individual transactions in the sub-ledger. In other words, after you enter, review, and approve transactions, you must post them to update your systems with current transaction records and maintain system integrity.

What is an SCD?

SCD stands for Slowly Changing Dimension, and is essentially a data structure that allows us to keep track of changes in our data.

What does “data change” mean?

In the world of IT, we typically store data in relational and non-relational databases. In this article, we start with relational databases, which are typically stored in SQL databases. Data entry is usually done through some kind of application where users input data using a desktop or sometimes some mobile device. No matter how, but the data eventually ends up in a data table. Consider a typical transaction: a sale event occurs and the following data (also called a record in a database language), that is, records are created

A data changes occur when a change, modification or deletion occurs in the already created and stored data (in this case the data above). SCD types are intended to symbolize the handling of the above changes.

More info on SCD:
https://www.datawarehouse4u.info/SCD-Slowly-Changing-Dimensions.html

https://www.allthingssql.com/slowly-changing-dimension-types/

Slowly changing dimension – Wikipedia

For example, SCD0 means that no changes are allowed to the previous data – (in practice, this is how transactions are stored in Block chain technologies, where you cannot delete or modify previous transaction data).

SCD1 is when we pass all changes to the current data file and do not store anywhere what data was stored before. Technically, SCD1 can already be considered a historization.

In this post, we show one of the benefits of SCD2 and how this type of historization can be used to create appropriate monthly posting files or a proper interface for accounting. You can read more about SCD2 and the different SCD formats following the links above

The task: Create a posting file with retroactive – aka. closed periods adjusting

Let’s suppose our company builds a calculation engine (hereinafter: the engine) in a database that receives transaction data from an application and then recalculates the transaction values from the received transaction data each time it runs.

( It doesn’t matter if this is a separate database, a database schema, or part of a transaction system to be developed, this solution is feasible in any case. I will now describe a solution where the transaction data is transferred using ETL processes to another database, and some of the calculations take place there, not in the application itself )

Several transformations can occur in the database that do not happen on the application side, e.g. Exchange rate assignment, assignment of partner data, product info, even purchase data, purchase prices, etc. On every run the engine receives the data from the application and from all other source systems and completely re-calculates the output data.

In our case, it is also the job of the engine to produce a posting file (or an interface) for the accounting program at an unspecified time at the beginning of each month. This part of the process is done manually, there is no automated running and data extraction

The Challenge

The engine completely recalculates everything on each run – clears the entire table contents at the database level (TRUNCATE) and loads the calculated data completely (INSERT).

Posting, on the other hand, occurs once a month at an unspecified time, during which only a portion of the calculated data (typically data for the previous month) is loaded into the accounting software (or ERP), based on a predefined field containing the transaction date.

The business analysis revealed that some transaction data may change even months after the first recording (UPDATE).

We did not receive any information regarding the possibility of data deletion during the development, however, it cannot be completely ruled out from the engine’s point of view

So to sum up the problem: how can we ensure the consistency of the calculated current data and the accounting data at the analytical level in the long run?

Let’s see our Example:

The transaction system will be launched in January 2022, and will be showing the following transactions when they are imported on the 2nd of February:

On the 3rd or March 2022 (the time of the next monthly posting/import) the whole transaction data set will look like this:

During the import on 3rd of March 2022, according to the protocol, only the data with the February TransactionDate (aka. PostingDate) will be loaded into the accounting system, however, we can see in the example that the data line (record) with the ID 3 has been deleted and for the record with Id 2, the Quantity was changed from 50 to 48.

The Alternatives

During the posting (in the posting file), we can follow the general cancellation rules, ie we make a change by adding the value of the previously recorded transaction with a negative sign (we can also post a canceled transaction in this way), and then re-enter the value and data of the correct transaction.

If the accounting software or ERP allows it, it is advisable to record the fact of the cancellation, and to record the ID linking the posted item to the original transaction together with the accounting item. (In our case there are two options: it is logical to use an ID field for this, however, if for some reason this field is not available, the InvoiceNumber and ItemNo lines can clearly identify a record, so we could use those as well).

A possible solution would be if the accounting system would always receive the full calculation, but this would cause a lot of data redundancy that would only make data reconciliation between the accounting and the engine even more difficult.

So keep in mind to make it as easy as possible to reconcile the differences.

The Solution: Historization and SCD2 helps!

First of all, we introduce historization into the engine in at least SCD2 format. During historization, it is sufficient to historize the final table containing each calculation step.

Historization – illustrated with data below – is done by introducing 2 Flag fields as well as two date fields, a DAT_FROM and a DAT_TO field, which show the validity period of the given transaction record.

The two flags are FLG_IS_CURRENT and FLG_IS_DELETED. Flags are managed as Bits, so they can only take a value of 0 or 1, 0 is false and 1 is true. (This form of SCD2 is just an option, no obligation for the data types, nor for the column names, etc. Try to grasp only it’s logic, if you are not familiar with it.

For the historization you will need to select the key field or key fields, using which the items can be clearly identified. These key fields determine whether an item was previously included in the data or not, and the modification steps are also performed based on these key fields.

When entering the system the current data will get the current date / time in the DAT_FROM field, the value of FLG_IS_CURRENT will be 1, the value of FLG_IS_DELETED = 0, and the value of the DAT_TO field will either be empty, ie NULL or e.g. 2999-12-31.

(I suggest a preferably high default value for the DAT_TO field, because NULL’s are mostly incomparable in rdbms systems.)

If an item is deleted, FLG_IS_CURRENT will be 0, FLG_IS_DELETED will be 1, and the DAT_TO value field will be updated to the time of the change.

If modified, the value of the FLG_IS_CURRENT field for the modified record changes to 0, and the value of DAT_TO changes to the time the change is registered (when the modified field is historized), and the field with new values is entered as the current new value in the historized table.

It is a good idea to store the current and historical tables in separate tables.

In the above case the HISTORY table would look like this on the 3rd of March 2022:

We can see that both items from January, which were modified, were modified in February, after the date of the last dispatch/import (February 2)

Preparing for posting/import

In order to be able to correct the posting with previously deleted or modified items, whether we export them manually or automatically for posting, we must record the date of the export. Let’s call this the DateOfExport parameter.

It is advisable to organize the actual solution by supplementing the record files containing the current transactions together with the plus sets needed for the corrections (and providing the sets with the appropriate signs), which can be handled to perform all the modification and transformation steps in the engine together which helps to produce the posting file.

The following figure will help you understand which sets to add to the current one for this purpose:

Explanation:

The blue and red areas are the ones that output the entire dataset that needs to be transformed.
The blue areas are the ones that are counted with their original sign, the red ones are calculated with the opposite sign.
It can be seen that only part of the entire ACTUAL file, is exported. The export is based on the original PostingDate field of the transaction data, that is, when the transaction was created.
The @LastDateOfExport is the date when previous month’s data was exported.
In addition to the current monthly data of the current record file, we need to add three sets of data:
- Deleted records: Records for which the following conditions are true in the historized table (each condition must be met, i.e. there is an “and” relationship between the conditions):
  - FLG_IS_CURRENT = 0
  - FLG_IS_DELETED = 1
  - DAT_TO › @LastDateOfExport – this guarantees that items that have been deleted AFTER a previous export will now be entered with a negative sign in the record set before the transformation.
- Historical data for modified records with a negative (opposite) sign: these are the records for which the following conditions are true in the historized table (each condition must be met, so there is an “and” relationship between the conditions):
  - FLG_IS_CURRENT = 0
  - FLG_IS_DELETED = 0
  - the value of @LastDateOfExport is between the DAT_FROM and DAT_TO records
- Current data of modified records: all records from the entire current set whose identifiers are included in the records defined in set 2. These are not duplications because they only contain records that were already current in a previous post, but their DAT_TO is greater than their @LastDateOfExport. The sign of the values of the present set is not changed.
One important modification: for all non-current records, the PostingDate value, ie the date value based on which it is selected for the given monthly post export, is changed to DAT_TO, thus guaranteeing that the added sets are included in the actual post file

The base record set generated this way:

We handle the above sets as a single input and we perform the transformations that are required for the sets to be accepted by the accounting software or ERP.

This blog entry does not cover this topic because these transformation are all target system specific and do not affect the data set and historization operations described above.

Benefits of this approach vs. other approaches

We don’t do a complete data file load into the accounting software or ERP system every time we do a posting. We do not burden the accounting systems with unnecessary records, simplifying the reconciliation.
We ensure continuous matching at the analytical level between the accounting software / ERP system and the reports showing the current data (eg towards partners)
We facilitates analytical reconciliations — especially when IDs are included in the posting file so that cancellation events (items recorded with the opposite sign) can be marked separately. This also facilitates coordination with partners and monthly and annual closing tasks.
We can also use this in case of automated connection. The only task is to store the DateOfExport value (recordeed manually in the present case study) in the “engine” after the successful posting.
We cover almost every possible business need creating a robust solution that is prepared for future requests. For example, even if initially there has been no demand to manage cancellations, if such requirement may arise later the engine will be prepared for this

Just one last thing:

Our Rapid solutions (Rapid Planning and Rapid Analytics) provide the ability to use all of the slowly change data management methodologies at the data warehouse level, so our company and our clients can easily create data market solutions supplemented with historization, based on which “time-machine” tabular models can be created. For example our datawarehouse automation solution (Rapid Analytics) is ideal for default historicization solutions

Thank you for your time! I hope you liked my post and found it useful. If you have any comments, additions or questions feel free to contact me at kreiszzs@abylon.hu

The post How to post G/L records with retroactive data with the help of BI and SCD? appeared first on Abylon.

Railway data map visualizations

Zoltán Zicherman — Mon, 11 Oct 2021 15:29:12 +0000

In the world of Power BI, there are no identical solutions. Both, the business requirements and the data are almost always different. In the ETL processes as well as in data modeling, we need to find the solutions that can provide the answers to the questions in the simplest and fastest way.

The complexity of the data processing, the running time, the transparency of the tabular model, the complexity of the measures used and the running time are also very important metrics measuring the quality of our work.

When visualizing railway data on a map, the appropriate model can be identified knowing the business meaning of the data and its volume and complexity. Typically, there are two types of data available in case of line-based infrastructures:

Point
Route (Railway route

Connecting the fact data with the map objects

Point data can be an extraordinary event (eg accident), data related to an infrastructure element (maximum speed, condition, scheduled maintenance, cost) or detailed traffic data for a certain train based on its railml schedule and actual traffic data. This data can be fitted to a star schema, and the dimension table serving the map can be linked to the model with a simple One to Many connection

In the image below, the EU_OBJECTS dimension table is the display source of the map, it will display the objects (line or point) broken down by ID, based on their type. The EU_TRANSPORT table contains the factual data:

Note: Abylon MAP requires the usage of a unique ID and and object type, so it can be mapped to geojson files, or individual lines can be displayed separately by direction. Click on the button below for more details.

Organizing routing or interval type data into a model

Traffic data that contains a measured value along its entire route is called reference data. Tihis includes passenger traffic, ticket sales data, travel from A to B, or train data that do not change from the start point to the end (eg weight, railway company), but do not include traffic time data or values calculated from them.

We have several choices regarding this data. The problem is similar to period-type data, and there are three available options:

Interval data is broken down into the smallest point-like data required:

Like from time interval to days, and then from relations to line segments and stations. The days of the time interval can be extracted into separate rows in the fact table and these (as points) can be added into the data model.

Tabular On the fly:

We entrust the data model to sort out the data using one or two connection tables. See later.

DAX on-the fly:

When working with small amounts of data, instead of complicated modeling or significant data multiplication, we can filter using a DAX filter, in which case the dimension and fact tables are not linked within the model. Continuing with the time interval example: calculate ([something],filter(’Fact’,[datfrom]<=max(’date’[date]) && [datto]>=min(’date’[date]))

Combining solutions 1 and 2 to find a calculation optimum.

In this case the fact data rows are not broken down to an elementary point / section level, but to a higher level (e.g. railway line) and the model itself remains in accordance with number 2. (author’s note: in my opinion, this is the best method, we can provide detailed, pin-level data with it, but more frequent queries tailored to business needs can occur faster).

We will now discuss solution number 2 in more detail.

Tables underlined in yellow color provide the dimensional expansion:

The row of the fact table contains data for complete routes, to which is attached the routes M2M table, which contains the unique values of the route identifiers.

Routes are connected with a many-to-many relationships to the MAP object level, whose final dimension table contains the map data: coordinates, names, and so on.

The Fact data is like this:

ID	FROM	TO	DIRECTION_ID
123	A	D	23
124	K	O	24

The directions M2M connection:

DIRECTION_ID

The MAP technical table is for extracting the directions:

DIRECTION_ID	MAP_OBJECT_ID
123	A
123	B
123	C
123	D
124
124
124

This approach allows you to assign data to an elementary map object without multiplying rows.

So, if we filter for an individual object in the MAP’s dimension (e.g an elementary section of line), the engine will calculate all of directions that have this element in the route description, and the fact data will be filtered for them

Which approach is the best?

In the case of point data, the question seems clear, although we have already seen a case where a set of factual data was related to a railway station and also to the related section. With railway data models, it is particularly interesting that you need to handle the relation to a railway line itself, so the master data of the map should contain the stations that belong to several lines multiple times.

In terms of where to start with relative/related data, the quantity and quality of the data is the decisive factor. With that knowledge (and clear business requirements) an experienced data modeler can easily chose the right approach.

If we set aside the railroad example and look for a more straightforward example, successful management of the time interval may still be the right example

Business Case	Solution
Short time intervals (1-3 days)	It is recommended to transform the time interval to a point regardless of the amount of data, as it does not cause an increase in the size of the rows of factual data.
Long time intervals with limited data (such as contract records)	In this case, the model-based on-the-fly and the daily breakdown would result in unnecessary duplication of rows, but the the dax on-the-fly approach is a good choice and we can treat the factual data with interval-filtered measures based on an independent date table.
Medium / variable time intervals, large amount of data (hotel reservations)	If our calculations show that breaking down the data into days would increase the number of rows by an order of magnitude (minimum 5-10 day average time interval), it is recommend to use the model-level on-the-fly approach.
Medium / variable time intervals, with extremely large amount of data or special requirements	In some cases, it is necessary to try to keep the size of the tables implementing M2M connections manageable. In this case it may be worth combining the different solutions: the lines are broken down into weeks we create explanatory tables for the weeks and for the combinations within a week. In this case, there will be 28 rows for a week, which can provide a favorable breakdown even for a multi-year interval.

Custom Power BI map visualizations for railway companies

Abylon MAP is built on top of Power BI offering, custom, advanced map visualization features for companies dealing with line-based infrastructures. It is ideal for railway (or other transportation companies dealing with line-based infrastructures) looking for an affordable, customizable, easy-to-use map visualization tool for Power BI.

Infrastructure-specific Power BI map visualizations
Visualization of line-based networks using both circles and lines
Full integration with Power BI (built on top of it)
Active maps based on Power BI models or GeoJSO

The post Railway data map visualizations appeared first on Abylon.

Multi-level access rights in Dynamic Row Level Security

Zoltán Zicherman — Fri, 13 Aug 2021 13:22:05 +0000

In today’s blog post we will discuss in detail the implementation of hierarchical dynamic RLS with Power BI without many-to-many connections.

Case Description

Our client’s request was to implement a dynamic RLS. We had proper table specifying username and authorization pairings/access rights. Based on the business needs a hierarchical management had to be implemented, something like this:

Country
- City
- Employee

In the fact tables each row includes a worker ID and a city ID. It is geographically assured that a certain city belongs to a certain country within the defined period. Based on the client’s request the rules for the visibility of fact and dimensional data are:

Senior managers can see the data of one or more country’s data (and their cities) and the data of the workers who work there
Area managers and sales manegers can see the data of one or more cities and the workers who work there
Employees can see their own data
There are administrators who can see everything
There may be employees who, apart from seeing their own data, can also see data generated in a city (eg sales and area manager at the same time).

This is the summary what we see in the logic that serves the RLS:

The problem

The solution described on the Microsoft site is pretty and nice, but does not solve the issues highlighted and underlined in the rules described above (in the previous paragraph).

Source: https://docs.microsoft.com/hu-hu/power-bi/connect-data/desktop-tutorial-row-level-security-onprem-ssas-tabular

The occasionally necessary many-to-many connections are easily doable using intermediate tables, but this approach is far from ideal in terms of modeling and performance.
A special problem are the users who “appear” in two categories/on two levels (the last two rows in the table above), because in their case you need to filter two dimensional tables, but due to the AND relationship of the filters, the user permission are not extended, but narrowed down, so instead of having a combination of the two data sets you only get a section of it.

The extended model (seen below) shows that an additional problem is the visibility of related dimension data, for example, the cities or countries dimension does not filter employee data, i.e. it is not guaranteed that a manager with city-level authority will only see the master data for employees who worked in his city

It is guaranteed that you will be lost among the various many-to-many switchboards and even if you do it, still:

• It is complicated
• therefore, it is difficult to test
• any corrections or modification carry a huge risk

The solution

Principles

Leaving behind our habits and past modeling practices, we have laid down new principles that provide a solution to the business need without over-complicating the model or the whole process:

We used a single dimension table to filter based on fact RLS conditions: Country, City, Employee
We used a single table containing user data to define the RLS: UserId, Access level etc.
Given that the number of returned value sets is low (filter for less than 10 country / city / employee IDs), we allowed ourselves to use DAX filters instead of m2m relationships in the model.

Preparation

Fact and dimension

You need to generate an unique identifier in the fact and dimension tables that ensures the transfer of RLS filters, which ensure that a city can belong to a country, so that the employee ID and the city will be connected:

concat([employee_id],’-’,[City_ID])

The source of the dimension table can be the fact table queried using distinct, or in the case of a corresponding HR strain, an extract of it. It is important that an identifier appears only once in the new dimension table and that the table must contain the data to be used in the report, eg name, telephone number, city coordinates, etc.

In case of more detailed demand it can be supplemented with validity management, and a period identifier can be appended to the identifier.

Viewed in PowerQuery, our dimension table will look like this:

The RLS table will be the summary table shown at the beginning:

Based on this the new model will be much more simple and clear

The new model contains:

all RLS data
all employee data
every city
data for all countries
the factual data to be examined, with a simple relationship

DAX Role Filtering

Basis

We assume that the access levels have separate AD groups, so the Role containing 3 filters can be added separately (at the end we will examine what if they don’t).

Logic

Filtering for Country:

The dimension table must be filtered for the rows that contain the country IDs that appear next to the user ID in the RLS table:

Conversely:

Collecting the country ID-s related to the users from the RLS

var usercountries=calculatetable(values(’RLS’[COUNTRY_ID]), ’RLS’[AD]=username()))

Dimension filtering for these ID-s:

filter(’Dim Employee-City’, ’Dim Employee-City’[COUNTRY_ID] in usercountries)

Implementation

Rules added to the Dim Employee-City table.

Country:

’Dim Employee-City’[COUNTRY_ID] in calculatetable(values(’RLS’[COUNTRY_ID]), ’RLS’[AD]=username()))

City:

’Dim Employee-City’[CITY_ID] in calculatetable(values(’RLS’[CITY_ID]), ’RLS’[AD]=username()))

Employee:

’Dim Employee-City’[EMPLOYEE_ID] in calculatetable(values(’RLS’[EMPLOYEE_ID]), ’RLS’[AD]=username()))

If there are no separate AD groups

The three rules can be combined into one.

’Dim Employee-City’[COUNTRY_ID]

in calculatetable(values(’RLS’[COUNTRY_ID]), ’RLS’[AD]=username()))

’Dim Employee-City’[CITY_ID]

in calculatetable(values(’RLS’[CITY_ID]), ’RLS’[AD]=username()))

’Dim Employee-City’[EMPLOYEE_ID]

in calculatetable(values(’RLS’[EMPLOYEE_ID]), ’RLS’[AD]=username()))

Summary (Lessons learned)

Although it is basic rule that we solve problems in a model or data source preferring SQL, aiming for the simplicity in case of DAX formulas, here from the model’s usability point of view the complex solution is less resource intensive:

working with two tables instead of using a complex model
we ensure low result numbers during RLS filtering (eg it does not return the ID of all employees in the country, but only 1-2 country codes or 1-2 cities)
the model remains simple, stable
We were able to create a general solution for the customer and subsequently this RLS solution has become a standard for several other data sets with similar / identical dimensional resolution.

The lesson of the solution is that many complex problems can be solved within the Microsoft BI ecosystem, no matter what tool we are using (SQL, .NET, Tabular or DAX), we just don’t need be afraid to step out of our schemas, standards, and try to think “outside the box” to find the ideal solutions.

The post Multi-level access rights in Dynamic Row Level Security appeared first on Abylon.

BI – Abylon

Effective Master Data Management with the Abylon Rapid Platform

Effective Master Data Management with the Abylon Rapid Platform

Who needs Master Data Management?

Use case I. - Basic categorization

Use case II. - Tax report change

Use case III. - Integration of new ERP system

Use case IV. - Affiliates with different accounting systems

Downsides of the "old-school" approach to Master Data Management

Business Intelligence with adequate Master Data Management

Modern enterprise-grade MDM in just a couple of weeks

Author of the post:

Found this post interesting?

Follow us for more news and technical insights!

Other popular posts

3+1 reasons why you need an Advanced Analytics Platform

What is Advanced Business Intelligence or Advanced Analytics?

Why would anyone need an Advanced Analytics platform?

1. Business benefits

2. Technological modernization

3. Sustainability: lower and more dynamic costs

+1 The future of BI is in the cloud (or multi-cloud, or hybrid)

How to get started?

Author of the post

Abylon Consulting’s experience with railway BI and data projects

Railways - a world of diverse, complex and challenging BI and data projects

Abylon Consulting and the Hungarian Railways

Rail BI and data project delivered by Abylon

Near real-time data datawarehousing

JÉSTAT+ Route decomposition and display of ticket dat

START-TÁR Railway vehicle condition evaluation

An award-worthy solution

Successful rail BI cooperation

Found this post interesting?

Fore more interesting articles and news follow our social media channels!

Other Popupar posts

SSAS Tabular model management with the Abylon Rapid Platform

Intro

Development

Processing

Logging

Case Study: Advanced Analytics on top of a legacy on-prem DWH

Introduction

Business Challenge

Solution – Advanced Analytics Platform with Azure Synapse and Databricks

More dynamic data

Easier access, larger audience

Scalability and reduced infrastructure costs

Advanced Analytics Platform – how to create a dream framework for Python/R

Introduction

What are my essential goals?

Introducing the components of our Advanced Analytics Platform

Power BI (PBI)

Azure Databricks (DBX)

Azure Datalake Storage (ADLS)

Azure Data Factory (ADF) – optional

High-level design

Implementation

Azure tenant

Resource Group

DBX

ADLS

Connect ADLS with DBX

Power BI

ADF

Real-life example

Some more tips & tricks, know-how and best practice!

Still curious for more?

Author of the post

What clients say about Rapid Analytics?

Interview: What our clients say about our DWH Automation tool

How to post G/L records with retroactive data with the help of BI and SCD?

For whom this blogpost is meant for

Intro

What is posting in accounting?

What is an SCD?

What does “data change” mean?

The task: Create a posting file with retroactive – aka. closed periods adjusting

The Challenge

The Alternatives